Predicting total nucleic acid yield and dissection boundaries for histology slides

ABSTRACT

A method for determining tumor block sufficiency for generating one or more hematoxylin and eosin (H&amp;E) slides by assessing an expected yield of nucleic acids for tumor cells and determining a number of H&amp;E slide for satisfying a desired total nucleic yield is provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a Continuation-in-Part of U.S. application Ser. No.16/830,186, filed on Mar. 25, 2020, which is a Continuation-in-Part ofU.S. application Ser. No. 16/732,242, filed on Dec. 31, 2019, whichclaims priority to U.S. Provisional Application Ser. No. 62/787,047,filed Dec. 31, 2018, and is a Continuation-in-Part of U.S. applicationSer. No. 16/412,362, filed on May 14, 2019, which claims priority toU.S. Provisional Application No. 62/671,300 filed May 14, 2018, andclaims priority to U.S. Provisional Application Ser. No. 62/824,039,filed on Mar. 26, 2019, U.S. Provisional Application Ser. No.62/889,521, filed Aug. 20, 2019 and to U.S. Provisional Application Ser.No. 62/983,524, filed on Feb. 28, 2020, the entire disclosures of eachof which are hereby expressly incorporated by reference herein.

FIELD OF THE INVENTION

The present disclosure relates to determining tumor block sufficiencyfor generating one or more hematoxylin and eosin (H&E) slides and, moreparticularly, determining tumor block sufficiency by assessing anexpected yield of nucleic acids for tumor cells and determining a numberof H&E slide for satisfying a desired total nucleic yield is provided.

BACKGROUND

The background description provided herein is for the purpose ofgenerally presenting the context of the disclosure. Work of thepresently named inventors, to the extent it is described in thisbackground section, as well as aspects of the description that may nototherwise qualify as prior art at the time of filing, are neitherexpressly nor impliedly admitted as prior art against the presentdisclosure.

To guide a medical professional in diagnosis, prognosis and treatmentassessment of a patient's cancer, it is common to extract and inspecttumor samples from the patient. Visual inspection can reveal growthpatterns of the cancer cells in the tumor in relation to the healthycells near them and the presence of immune cells within the tumor.Conventionally, pathologists, members of a pathology team, other trainedmedical professionals, or other human analysts visually analyze thinslices of tumor tissue mounted on glass microscope slides and identifyeach region of the tissue as corresponding to one of many tissue typesthat are present in a tumor sample. This information aids thepathologist in determining characteristics of the cancer tumor in thepatient, which can inform treatment decisions. A pathologist will oftenassign one or more numerical scores to a slide, based on this visualapproximation.

To perform these visual approximations, medical professionals attempt toidentify a number of characteristics of a tumor, including, for example,tumor grade, tumor purity, degree of invasiveness of the tumor, degreeof immune infiltration into the tumor, cancer stage, and anatomic originsite of the tumor, which can be important for diagnosing and treating ametastatic tumor. These details about a cancer can help a physicianmonitor the progression of cancer within a patient and can helphypothesize which anti-cancer treatments are likely to be successful ineliminating cancer cells from the patient's body.

Another tumor characteristic is the presence of specific biomarkers orother cell types in or near the tumor, including immune cells. Forexample, tumor-infiltrating lymphocytes (TILs) present in elevatedlevels have been recognized as a biomarker of anti-tumor immune responseacross a wide range of tumors. TILS are mononuclear immune cells thatinfiltrate tumor tissue or stroma and have been described in severaltumor types, including breast cancer. A population of TILs is composedof different types of cells (i.e., T cells, B cells, Natural Killer (NK)cells, etc.) in variable proportions. The population of TILs naturallyoccurring in a cancer patient is largely ineffective in destroying atumor, but the presence of TILs has been associated with improvedprognosis in many cancer types, including, e.g., epithelial ovariancarcinoma, colon cancer, esophageal cancer, melanoma, endometrialcancer, and breast cancer (see, e.g., Melichar et al., Anticancer Res.2014; 34(3):1115-25; Naito et al., Cancer Res. 1998; 58(16):3491-4).

Yet another tumor characteristic is the presence of specific moleculesas a biomarker, including the molecule known as programmed death ligand1 (PD-L1). PD-L1 has been linked with diagnosing and assessing non-smallcell lung cancer (NSCLC), which is the most common type of lung cancer,affecting over 1.5 million people worldwide. NSCLC often responds poorlyto standard of care chemoradiotherapy and has a high incidence ofrecurrence, resulting in low 5-year survival rates. Advances inimmunology show that NSCLC frequently elevates the expression of PD-L1to bind to programmed death-1 (PD-1) expressed on the surface ofT-cells. PD-1 and PD-L1 binding deactivates T-cell antitumor responses,enabling NSCLC to evade targeting by the immune system. The discovery ofthe interplay between tumor progression and immune response has led tothe development and regulatory approval of PD-1/PD-L1 checkpointblockade immunotherapies like nivolumab and pembrolizumab. Anti-PD-1 andanti-PD-L1 antibodies restore antitumor immune response by disruptingthe interaction between PD-1 and PD-L1. Notably, PD-L1-positive NSCLCpatients treated with these checkpoint inhibitors achieve durable tumorregression and improved survival.

As the role of immunotherapy in oncology expands, accurate assessment oftumor PD-L1 status may be useful in identifying patients who may benefitfrom PD-1/PD-L1 checkpoint blockade immunotherapy. Currently,immunohistochemistry (IHC) staining of tumor tissues acquired frombiopsy or surgical specimens is employed to assess PD-L1 status.However, such IHC staining is often limited by insufficient tissuesamples and, in some settings, a lack of resources.

Hematoxylin and eosin (H&E) staining is a longstanding method used bypathologists to analyze tissue morphological features for malignancydiagnosis. H&E slides, for example, can illustrate visualcharacteristics of tissue structures such as cell nuclei and cytoplasm,to inform identification of cancer tumors.

Similar considerations for sufficiency in tissue samples exist for H&Eslides, as well as IHC slides. In some instances, complications frominsufficient tissue samples are exacerbated during subsequent sequencingof the available tissue samples when the total nucleic yield of thetumor tissue within a sample is insufficient to generate reliablesequencing results. While reliable analysis requires a sufficientquantity of nucleic yield from the tissue sample, all tissue samplesprovided to analysis, such as staining or sequencing, which are ofinsufficient quantity are lost due to the destructive natures ofextracting nucleic acid from the tissue samples and therefore an entirenew batch is required for processing to generate reliable results. Theseadditional batches of tissue samples, lost due to reprocessingadditional samples for staining or sequencing, further reduces theavailable tissue samples, and a patient may have to undergo anotherpainful and costly biopsy or surgical procedure to acquire sufficienttissue samples. There is a need for identifying the sufficiency of thenucleic acid yield within tumor tissue to prevent processing and wasteof attempting analysis of tissue with insufficient total nucleic yield.

Technological advances have enabled the digitization of histopathologyH&E and IHC slides into high resolution whole slide images (WSIs),providing opportunities to develop computer vision tools for a widerange of clinical applications. High-resolution, digital images ofmicroscope slides make it possible to use computer based analysis ofslides in the hopes of classifying tissue by type or pathology.Generally speaking, for example, deep learning applications have shownpromise as a tool in medical diagnostic applications and in predictingtreatment outcomes. Deep learning is a subset of machine learningwherein models may be built with a number of discrete neural nodelayers. A Convolutional Neural Network (“CNN”) is a neural network thatemploys convolution techniques. For example, a CNN may provide for adeep learning process that analyzes digital images by assigning oneclass label to each input image. WSIs, however, include more than onetype of tissue, including the borders between neighboring tissueclasses. There is a need to classify different regions as differenttissue classes, in part to analyze the borders between neighboringtissue classes and the presence of immune cells among tumor cells. For atraditional CNN to assign multiple tissue classes to one slide image,the CNN would need to separately process each section of the image thatneeds a tissue class label assignment. However, neighboring sections ofthe image overlap, such that processing each section separately wouldcreate a high number of redundant calculations and would be timeconsuming.

A Fully Convolutional Network (FCN) is another type of deep learningprocess. A FCN can analyze an image and assign classification labels toeach pixel within the image. As a result, compared to a CNN, a FCN canbe more useful for analyzing images that depict objects with more thanone classification. Some FCNs generate an overlay map to show thelocation of each classified object in the original image. However, to beeffective, FCN deep learning algorithms would require training data setsof images with each pixel labeled as a tissue class, and that requirestoo much annotation time and processing time to be practical. In adigital WSI image, each edge of the image may contain more than10,000-100,000 pixels. The full image may have at least 10,000² to100,000² pixels, which would force incredibly long algorithm run timesto attempt tissue classification. Simply put, the high number of pixelsmakes it infeasible to use traditional FCNs to segment digital images ofslides.

There is a need for new easily accessible techniques of diagnostictesting for biomarkers, such as TILs, PD-L1, and others using H&Eimages, for identifying and characterizing such biomarkers in anefficient manner, across population groups, for producing betteroptimized drug treatment recommendations and protocols, and improvedforecasting of disease progression.

SUMMARY OF THE INVENTION

The present application presents an image-based nucleic acid yieldprediction system formed of a deep learning framework configured andtrained to directly learn from histopathology slide images and predictthe yield of nucleic acid from the present tumor cells in medicalimages. In examples, deep learning frameworks are configured and trainedto analyze histopathology images and identify tumor cells and predict anexpected yield of nucleic acid for the tumor cells. In various examples,these deep learning frameworks are configured to include differenttrained tissue and/or cell classifiers each configured to receiveunlabeled histopathology images and provide yield of nucleic acid fromthe present tumor cells predictions for those images. These yield ofnucleic acid predictions may then be used to reduce a large set ofavailable immunotherapies to a reduced, small subset of targetedimmunotherapies that medical professionals may use to treat patients. Invarious examples, deep learning frameworks are provided that identifybiomarkers indicating the presence of a tumor, a tumor state/condition,or information about a tumor of the tissue sample, from which a set oftarget immunotherapies can be determined.

Cells contain nucleic acids (nucleotides) and analysis processes such assequencing often destroy cells during extraction of nucleic acids fromthe cells. As such, the yield of nucleic acids is a quantitativemeasurement of the nucleic acids which may be extracted from the cells.When this quantity falls below the operating capacity of a sequencingmachine to effectively analyze and generate reliable sequencing resultsfor the deoxyribonucleic acid (DNA) or ribonucleic acid (RNA) of thetissue, the extracted nucleic acids are lost and cannot be reused insubsequent attempts. Furthermore, the specific sequencing machine and/orthe type of sequencing results to be generated may change the totalyield of nucleic acids that are needed to generate reliable results.Therefore, the total nucleic acid yield may be identified based on acombination of the sequencing machine and/or the type of sequencing tobe performed and may be selected from, for example, 50-2000 ng. Assequencing machines become more efficient, the lower boundary of therange may be correspondingly decreased to the specification requirementsof the sequencing machine and as the type of sequencing analysis to beperformed changes, the upper bound may be correspondingly increased tothe requirements necessary for generating reliable results of the neweranalysis. In some instances, there may be additional tissue on the slidewhich is not tumor tissue which may affect the quality of the analysis.In these instances, a dissection boundary may be drawn around the tumortissue of the slide to provide a guide for extracting only the cellswhich should be sequenced to ensure the highest quality of analysis.This boundary reduces the total nucleic yield because there are fewercells provided for nucleic acid extraction. Estimates for the number ofslides to be scraped may be generated to prevent waste and ensure thatthe first attempt at analysis is successful. The dissection boundary maybe a microdissection boundary, a macrodissection boundary, or a wholeslide boundary. The expected yield of nucleic acid for the tumor cellswithin the dissection boundary may be predicted in various ways,including providing digital images of an H&E slide to a model trained ona H&E slides having labeled dissection boundaries and labeled totalnucleic yield, counting the number of tumor cells identified within thedissection boundary and multiplying the count by a tumor cell averagenucleic acid yield, calculating the surface area of the dissectionboundary and multiplying the surface area by a dissection boundaryaverage nucleic acid yield.

In examples, systems include deep learning frameworks trained to analyzefor and predict tumor status for histopathology images received fromnetwork-accessible image sources, such as medical labs or medicalimaging machines. As used herein, tumor status includes at least thenucleic acid yield from the tissue to be scrapped and analyzed. Inexamples, the techniques and systems herein may further generate reportsof predicted tumor status, e.g., yield of nucleic acid for tumor cells,which can then be stored, displayed, used for further operations, etc.These predicted yield of nucleic acid from the present tumor cellsreports may be provided to network-accessible systems, such as pathologylabs and primary care physician systems, for storage and display, and inuse in determining a cancer treatment protocol (i.e., immunotherapytreatment or chemotherapy treatment) for the patient. In some examples,the predicted yield of nucleic acid from the present tumor cellsdetermined at a pathology lab or primary care physician system or otherentity. In some examples, the predicted yield of nucleic acid from thepresent tumor cells reports may be input to network-accessible nextgeneration sequencing systems for driving subsequent genomic sequencing,or input to computerized cancer therapy decision systems for filteringtherapy listings down to tumor-determined matched therapies.

The techniques herein are capable of identifying tumor cells associatedwith any of a variety of cancers. Exemplary cancers include but are notlimited to, adrenocortical carcinoma, lymphoma, anal cancer, anorectalcancer, basal cell carcinoma, skin cancer (non-melanoma), biliarycancer, extrahepatic bile duct cancer, intrahepatic bile duct cancer,bladder cancer, urinary bladder cancer, osteosarcoma, brain tumor, brainstem glioma, breast cancer (including triple negative breast cancer),cervical cancer, colon cancer, colorectal cancer, lymphoma, endometrialcancer, esophageal cancer, gastric (stomach) cancer, head and neckcancer, hepatocellular (liver) cancer, kidney cancer, renal cancer, lungcancer, melanoma, cancer of the tongue, oral cancer, ovarian cancer,pancreatic cancer, prostate cancer, uterine cancer, testicular cancer,and vaginal cancer.

In some examples, the image-based nucleic acid yield prediction systemsare formed of deep learning frameworks having a multiscale configurationdesigned to perform classification on (labeled or unlabeled)histopathology images using classifiers trained to classify tiles ofreceived histopathology images. In some examples, the multiscaleconfigurations contain tile-level tissue classifiers, i.e., classifierstrained using tile-based deep learning training. In some examples, themultiscale configurations contain pixel-level cell classifiers and cellsegmentation models. In some examples, the classifications form thetile-level tissue classifiers and from the pixel-level cell classifiersare analyzed to predict biomarker status in the histopathology image. Inyet some examples, the multiscale configurations contain tile-levelbiomarker classifiers.

In some examples, the image-based nucleic acid yield prediction systemsare formed of deep learning frameworks having a single-scaleconfiguration designed to perform classifications on (labeled orunlabeled) histopathology images using classifiers trained usingmultiple instance learning (MIL) techniques. In some examples, thesingle-scale configurations contain slide-level classifiers trainedusing gene sequencing data, such as RNA sequencing data. That is,slide-level classifiers are trained using RNA sequence data to developimage-based classifiers capable of predicting biomarker status inhistopathology images.

In accordance with an example, a computer-implemented method includes:receiving a first hematoxylin and eosin (H&E) slide prepared from atumor block at an image-based nucleic acid yield prediction systemhaving one or more processors; identifying, using the one or moreprocessors, tumor cells of the H&E slide from a digital image;generating a tumor area mask based at least in part on the identifiedtumor cells, wherein the tumor area mask defines a dissection boundaryof the H&E slide associated with the digital image; predicting anexpected yield of nucleic acid for the tumor cells within the dissectionboundary; based on the predicted expected yield of nucleic acid,identifying a number of H&E slides to prepare from the tumor block tosatisfy a total nucleic yield; and preparing the number of H&E slidesfrom the tumor block when the predicted expected yield of nucleic acidexceeds a predetermined threshold.

In an example, the predetermined threshold is based at least in part onone or more specification requirements.

In an example, specification requirements are based on a minimum surfacearea within the dissection boundary. In an example, specificationrequirements are based on a minimum cell count within the dissectionboundary. In an example, specification requirements are based on aminimum cell density within the dissection boundary. In an example,specification requirements are set by an entity.

In an example, the predetermined threshold is selected from a rangebetween and including 50 ng-2000 ng.

In an example, predicting the expected yield of nucleic acid for thetumor cells within the dissection boundary includes: providing thedigital image of the H&E slide to a model trained on a plurality of H&Eslides having labeled dissection boundaries and labeled total nucleicyield.

In an example, predicting the expected yield of nucleic acid for thetumor cells within the dissection boundary includes: counting the numberof tumor cells identified within the dissection boundary and multiplyingthe count by a tumor cell average nucleic acid yield.

In an example, predicting the expected yield of nucleic acid for thetumor cells within the dissection boundary includes: calculating thesurface area of the dissection boundary and multiplying the surface areaby a dissection boundary average nucleic acid yield.

In an example, the dissection boundary is a microdissection boundary.

In an example, the dissection boundary is a macrodissection boundary.

In an example, the dissection boundary is a whole slide boundary.

In an example, identifying the tumor cells of the H&E slide from thedigital image includes identifying the tumor cells using a trained cellsegmentation model.

In an example, identifying the tumor using the trained cell segmentationmodel includes: applying, using the one or more processors, a pluralityof tile images formed from the digital image to the trained cellsegmentation model and, for each tile, assigning a cell classificationto one or more pixels within the tile image.

In an example, assigning the cell classification to one or more pixelswithin the tile image includes: identifying, using the one or moreprocessors, the one or more pixels as a cell interior, a cell border, ora cell exterior and classifying the one or more pixels as the cellinterior, the cell border, or the cell exterior.

In an example, the trained cell segmentation model is a pixel-resolutionthree-dimensional UNet classification model trained to classify a cellinterior, a cell border, and a cell exterior.

In an example, identifying the tumor cells using the trained cellsegmentation model includes: applying, using the one or more processors,each of a plurality of tile images formed from the digital image to thetrained cell segmentation model; and performing, using the one or moreprocessors, a registration on segmented cells in each of the tile imagesby determining a cell border of each cell, determining a centroid ofeach cell, and shifting coordinates of centroids to a universalcoordinate space for the digital image.

In an example, the trained cell segmentation model is trained using aset of H&E slide training images annotated with identified cell borders,identified cell interiors, and identified cell exteriors.

In an example, the method further includes superimposing, using aviewer, the tumor area mask over the digital image to visually indicateto a user which tumor cells to scrape.

In an example, generating the tumor area mask further includes providingthe digital image of the H&E slide to a model trained on a plurality ofH&E slides having dissection labels.

In an example, a system includes a pathology slide scanner systemconfigured to perform the methods herein.

In an example, the system is further configured to generate a digitalimage of each of the one or more H&E slides from the tumor block.

In an example, the pathology slide scanner system is communicativelycoupled to an image-based tumor cells prediction system through acommunication network for providing the one or more of digital images.

In an example, the one or more processors are one or more graphicsprocessing units (GPUs), tensor processing units (TPUs), and/or centralprocessing units (CPUs).

In an example, the expected yield of nucleic acid is confirmed throughsequencing the tumor cells within the dissection boundary.

In an example, sequencing is next-generation sequencing.

In an example, sequencing is short-read sequencing.

BRIEF DESCRIPTION OF THE DRAWINGS

This patent or application file contains at least one drawing executedin color. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the United States Patent andTrademark Office upon request and payment of the necessary fee.

The figures described below depict various aspects of the system andmethods disclosed herein. It should be understood that each figuredepicts an example of aspects of the present systems and methods.

FIG. 1 is a block diagram of a schematic of a prediction system havingan imaging-based biomarker prediction system, for example, as may beimplemented as an image-based nucleic acid yield prediction system, inaccordance with an example.

FIG. 2 is a block diagram of process for a conventional pathologistcancer diagnosis workflow.

FIG. 3 is a block diagram of a schematic of a deep learning frameworkthat may be implemented in the system of FIG. 1, in accordance with anexample.

FIG. 4 is a block diagram of a schematic of a machine learning dataflow, in accordance with an example.

FIG. 5 is a block diagram of a schematic of a deep learning frameworkformed a plurality of different marker classification models, as may beimplemented in the systems of FIG. 1 and FIG. 3, in accordance with anexample.

FIG. 6 is a block diagram of a process for imaging-based biomarkerprediction, for example, as may be implemented for image-based nucleicacid yield prediction, in accordance with an example multiscaleconfiguration.

FIG. 7 is a block diagram of an example process for determining apredicted biomarker status, in accordance with an example implementationof the process of FIG. 6, in accordance with an example.

FIG. 8 is a block diagram of a process for imaging-based biomarkerprediction, in accordance with an example single-scale configuration.

FIG. 9 is a block diagram of a process for generating a biomarkerprediction report and overlay map as may be performed by the systems ofFIGS. 1 and 3, in accordance with an example.

FIGS. 10A and 10B illustrate example overlay maps generated by theprocess of FIG. 9, showing a tissue overlay map (FIG. 10A) and a celloverlay map (FIG. 10B), in accordance with an example.

FIG. 11 is a block diagram of a process for preparing digital images ofhistopathology slides for classification, in accordance with an example.

FIGS. 12A-12C illustrate example neural network architectures that maybe used for classification models, in accordance with an example.

FIG. 13 illustrates a histopathology image showing tile images forclassification, in accordance with an example.

FIG. 14 is a block diagram of a schematic of an imaging-based biomarkerprediction system that employs separate pipelines, for example, as maybe implemented as an image-based nucleic acid yield prediction system,in accordance with another example.

FIG. 15A is a block diagram of a schematic of an example biomarkerprediction process as may be implemented by the system of FIG. 14, forexample, as may be implemented as an image-based nucleic acid yieldprediction system, in accordance with an example.

FIG. 15B is a block diagram of a schematic of an example trainingprocess as may be implemented by the system of FIG. 14, in accordancewith an example.

FIGS. 16A-16F illustrate input histopathology images, in accordance withan example. FIGS. 16A-16C illustrate a representative PD-L1 positivebiomarker classification example. FIG. 16A displays an input H&E image;FIG. 16B displays a probability map overlaid on the H&E image: FIG. 16Cdisplays a PD-L1 IHC stain for reference. FIGS. 16D-16F illustrate arepresentative PD-L1 negative biomarker classification example. FIG. 16Ddisplays an input H&E image; FIG. 16E displays a probability mapoverlaid on the H&E image; and FIG. 16F displays a PD-L1 IHC stain forreference. The color bar indicates the predicted probability of thetumor PD-L1+ class.

FIG. 17 is a block diagram of an example multi-field of view strategyfor PD-L1 classification as may be performed by the processes of FIGS.14, 15A, and 15B, in accordance with an example.

FIG. 18 is a block diagram of a schematic machine learning architecturehaving capable of performing label-free annotation training of a deeplearning framework and having a multiple instance learning controller,in accordance with an example.

FIGS. 19, 20, 21, and 22 are block diagrams of a framework operationthat may be implemented by the multiple instance learning controller inFIG. 18, in accordance with an example.

FIG. 23 is an example resulting overlap map showing biomarkerclassification for CMS, in accordance with an example.

FIG. 24 is block diagram of another framework operation that may beimplemented by the multiple instance learning controller in FIG. 18, inaccordance with another example.

FIG. 25 is an example resulting overlap map showing biomarkerclassification for CMS, in accordance with another example.

FIG. 26 is block diagram of another framework operation that may beimplemented by the multiple instance learning controller in FIG. 18, inaccordance with another example.

FIG. 27 illustrates an example neural network architecture that may beused for classification models, in accordance with another example.

FIG. 28 is a block diagram of a process for determining a listing ofmatched potential therapies, such as immunotherapies, in accordance withan example.

FIG. 29 is a block diagram of data flow for generating a listing ofmatched potential therapies, in accordance with an example.

FIG. 30 is a block diagram of a system for performing imaging-basedbiomarker prediction along with a pathology scanner system, for example,as may be implemented as an image-based nucleic acid yield predictionsystem, in accordance with an example.

FIGS. 31-37 illustrate various screenshots of generated graphic userinterfaces displays as may be generated by systems such as the systemsof FIGS. 1, 3, and 30, in accordance with an example.

FIG. 38 is a block diagram of an example computing device for use inimplementing various systems herein, in accordance with an example.

FIG. 39 is a block diagram of data flow for generating a dissectionboundary and predicting number of slides, in accordance with an example.

FIG. 40 is an illustration of a Smart Pathology pipeline for generatingdissection boundaries, estimating dissection yields, and predicting anumber of slides, in accordance with an example.

FIG. 41 is an illustration of a Smart Pathology generated graphical userinterface display as may be generated by systems such as the systems ofFIGS. 1, 3, 30, and 40 in accordance with an example.

DETAILED DESCRIPTION

An imaging-based biomarker prediction system is formed of a deeplearning framework configured and trained to directly learn fromhistopathology slides and predict the presence of biomarkers in medicalimages. The deep learning frameworks may be configured and trained toanalyze medical images and identify biomarkers that indicate thepresence of a tumor, a tumor state/condition, or information about atumor of the tissue sample.

In an implementation, a cloud-based deep learning framework is used formedical image analysis. Deep learning algorithms automatically learnsophisticated imaging features for enhanced diagnosis, prognosis,treatment indication, and treatment response prediction. In examples,the deep learning frameworks are able to directly connect to cloudstorage and leverage resources on cloud platforms for efficient deeplearning algorithm training, comparison, and deployment.

In some examples, the deep learning frameworks include a multiscaleconfiguration that uses a tiling strategy to accurately capturestructural and local histology of various diseases (e.g., cancer tumorprediction). These multiscale configurations perform classification on(labeled or unlabeled) histopathology images using classifiers trainedto classify tiles of received histopathology images. In some examples,the multiscale configurations contain tile-level tissue classifiers,i.e., classifiers trained using tile-based deep learning training. Insome examples, the multiscale configurations contain pixel-level cellclassifiers and cell segmentation models. In some examples, theclassifications form the tile-level tissue classifiers and from thepixel-level cell classifiers are analyzed to predict biomarker status inthe histopathology image. In yet some examples, the multiscaleconfigurations contain tile-level biomarker classifiers. Once trailed,the multiscale classifiers can receive a new labeled or unlabeledhistopathology image and predict the presence of certain biomarkers inthe associated histopathological slide.

In some examples, the deep learning frameworks herein include asingle-scale configuration trained using a multiple instance learning(MIL) strategy to predict biomarkers presence in histopathology images.A classifier trained using a single-scale configuration may be trainedto perform classifications on (labeled or unlabeled) histopathologyimages using classifiers trained using one or more multiple instancelearning (MIL) techniques. In some examples, the single-scaleconfigurations contain slide-level classifiers trained using genesequencing data, such as RNA sequencing data, and trained to analyzehistopathology images having slide-level labels, not tile-level labels.That is, slide-level classifiers are trained using RNA sequence data todevelop image-based classifiers capable of predicting biomarker statusin histopathology images.

Any of the multiscale and single-scale configurations herein mayincorporate various algorithmic optimizations to accelerate computationfor such disease analysis.

In an implementation of a multiscale classifier configuration, a deeplearning framework may be trained to include classifiers that performautomatic cell segmentation, determine cell/biomarker type, anddetermine tissue type classification from histopathology images therebyproviding image-based biomarker development. Even single-scaleclassifiers may be trained to include tissue type classification andbiomarker classification.

For multiscale classifier configurations, for example, aggregate andspatial imaging features concerning different cell types (e.g. tumor,stroma, lymphocyte) in digital hematoxylin & eosin (H&E) slides may bedetermined by a deep learning framework and used to predict clinical andtherapeutic outcomes. In place of rudimentary manual cell typeclassification, examples herein employ multiscale configurations in deeplearning frameworks to classify each sub-region of an H&E slidehistopathology image into a specific cell segmentation, cell type, andtissue type. From there, biomarker detection is performed by anotherdeep learning framework configured to identify various types of imagingmetrics. Example imaging metrics include tumor shape, including tumorminimum shape and max shape, tumor area, tumor perimeter, tumor %, cellshape, including cell area, cell perimeter, cell convex area ratio, cellcircularity, cell convex perimeter area, cell length, lymphocyte %,cellular characteristics, cell textures, including saturation,intensity, and hue.

Examples of tissue classes include but are not limited to tumor, stroma,normal, lymphocyte, fat, muscle, blood vessel, immune cluster, necrosis,hyperplasia/dysplasia, red blood cells, and tissue classes or cell typesthat are positive (contain a target molecule of an IHC stain, especiallyin a quantity larger than a certain threshold) or negative for an IHCstain target molecule (do not contain that molecule or contain aquantity of that molecule lower than a certain threshold).

In some examples, biomarker detection may be enhanced by combiningimaging metrics with structured clinical and sequencing data to developenhanced biomarkers.

Biomarkers may be identified through any of the following models. Anymodels referenced herein may be implemented as artificial intelligenceengines and may include gradient boosting models, random forest models,neural networks (NN), regression models, Naive Bayes models, or machinelearning algorithms (MLA). A MLA or a NN may be trained from a trainingdata set. In an exemplary prediction profile, a training data set mayinclude imaging, pathology, clinical, and/or molecular reports anddetails of a patient, such as those curated from an EHR or geneticsequencing reports. MLAs include supervised algorithms (such asalgorithms where the features/classifications in the data set areannotated) using linear regression, logistic regression, decision trees,classification and regression trees, Naïve Bayes, nearest neighborclustering; unsupervised algorithms (such as algorithms where nofeatures/classification in the data set are annotated) using Apriori,means clustering, principal component analysis, random forest, adaptiveboosting; and semi-supervised algorithms (such as algorithms where anincomplete number of features/classifications in the data set areannotated) using generative approach (such as a mixture of Gaussiandistributions, mixture of multinomial distributions, hidden Markovmodels), low density separation, graph-based approaches (such as mincut,harmonic function, manifold regularization), heuristic approaches, orsupport vector machines. NNs include conditional random fields,convolutional neural networks, attention based neural networks, deeplearning, long short term memory networks, or other neural models wherethe training data set includes a plurality of tumor samples, RNAexpression data for each sample, and pathology reports covering imagingdata for each sample. While MLA and neural networks identify distinctapproaches to machine learning, the terms may be used interchangeablyherein. Thus, a mention of MLA may include a corresponding NN or amention of NN may include a corresponding MLA unless explicitly statedotherwise. Training may include providing optimized datasets, labelingthese traits as they occur in patient records, and training the MLA topredict or classify based on new inputs. Artificial NNs are efficientcomputing models which have shown their strengths in solving hardproblems in artificial intelligence. They have also been shown to beuniversal approximators (can represent a wide variety of functions whengiven appropriate parameters). Some MLA may identify features ofimportance and identify a coefficient, or weight, to them. Thecoefficient may be multiplied with the occurrence frequency of thefeature to generate a score, and once the scores of one or more featuresexceed a threshold, certain classifications may be predicted by the MLA.A coefficient schema may be combined with a rule-based schema togenerate more complicated predictions, such as predictions based uponmultiple features. For example, ten key features may be identifiedacross different classifications. A list of coefficients may exist forthe key features, and a rule set may exist for the classification. Arule set may be based upon the number of occurrences of the feature, thescaled weights of the features, or other qualitative and quantitativeassessments of features encoded in logic known to those of ordinaryskill in the art. In other MLA, features may be organized in a binarytree structure. For example, key features which distinguish between themost classifications may exist as the root of the binary tree and eachsubsequent branch in the tree until a classification may be awardedbased upon reaching a terminal node of the tree. For example, a binarytree may have a root node which tests for a first feature. Theoccurrence or non-occurrence of this feature must exist (the binarydecision), and the logic may traverse the branch which is true for theitem being classified. Additional rules may be based upon thresholds,ranges, or other qualitative and quantitative tests. While supervisedmethods are useful when the training dataset has many known values orannotations, the nature of EMR/EHR documents is that there may not bemany annotations provided. When exploring large amounts of unlabeleddata, unsupervised methods are useful for binning/bucketing instances inthe data set. A single instance of the above models, or two or more suchinstances in combination, may constitute a model for the purposes ofmodels, artificial intelligence, neural networks, or machine learningalgorithms, herein.

In some examples, the present techniques provide for machine learningassisted histopathology image review that includes automaticallyidentifying and contouring a tumor region, and/or characteristics ofregions or cell types within a region (for example, lymphocytes, PD-L1positive cells, tumors having a high degree of tumor budding, etc.),counting cells within that tumor region, and generating a decision scoreto improve the efficiency and the objectivity of pathology slide review.

As used herein, the term “biomarkers” refers to image-derivedinformation relating to the screening, diagnosis, prognosis, treatment,selection, disease monitoring, progression, and disease reoccurrence ofcancer or other diseases, and in particular information in the form ofmorphological features identifiable in histologically stained samples.The biomarkers herein may be of morphological features determined fromlabeled based images in some examples. The biomarkers herein may be ofmorphological features determined from labeled RNA data. References to“biomarkers” herein is intended to include tumor cells and an expectedyield of nucleic acid for tumor cells. Image-derived information hereinfurther includes the total nucleic acid yield of tissue, withinrespective histology samples.

The biomarkers herein may be image-derived information that iscorrelated with the existence of cancer or of a susceptibility to cancerin the subject; the likelihood that the cancer is one subtype vs.another; the presence or proportion of biological characteristics, suchas tissue, cellular, or protein types or classes; the probability that apatient will or will not respond to a particular therapy or class oftherapy; the degree of the positive response that would be expected fora therapy or class of therapies (e.g., survival and/or progression-freesurvival); whether a patient is responding to a therapy; or thelikelihood that a cancer will regress, has progressed, or will progressbeyond its site of origin (i.e., metastasize).

Example biomarkers predicted from histopathology images using thevarious techniques herein include the following.

Tumor-infiltrating lymphocytes (TILs), as used herein, refers tomononuclear immune cells that infiltrate tumor tissue or stroma. TILsinclude, e.g., T cells, B cells, and NK cells, populations of which canbe subcategorized based on function, activity, and/or biomarkerexpression. For example, a population of TILs may include cytotoxic Tcells expressing, e.g., CD3 and/or CD8, and regulatory T cells (alsoknown as suppressor T cells), which are often characterized by FOXP3expression. Information regarding TIL density, location, organization,and composition provide valuable insight as to prognosis and potentialtreatment options. In various aspects, the disclosure provides a methodof predicting TIL density in a sample, a method of distinguishingsubpopulations of TILs in a sample (e.g., distinguishingCD3/CD8-expressing cytotoxic T cells from FOXP3 Tregs), a method ofdistinguishing stromal versus intratumoral TILs, and the like.

Programmed death-ligand 1 (PD-L1) is a 40 kDa type 1 transmembraneprotein that plays a role in suppressing the immune system, particularlyaffecting patients with autoimmune disease, cancer, and other diseasestates. Of relevance for cancer immunotherapy, PD-L1 can be expressed onthe surface of tumor cells, tumor-associated macrophages (TAMs), and Tlymphocytes and can subsequently inhibit PD-1-positive T cells.

Ploidy refers to the number of sets of homologous chromosomes in thegenome of a cell or an organism. Examples include haploid which meansone set of chromosomes and diploid means two sets of chromosomes. Havingmultiple sets of paired chromosomes in a genome of an organism isdescribed as polyploid. Three sets of chromosomes, 3n, is triploidwhereas four sets of chromosomes, 4n, is tetraploid. Extremely largenumber of sets may be designated by number (for example 15-ploid forfifteen sets).

Nucleus-to-cytyoplasm (NC) ratio is a measurement of the ratio of thesize of a nucleus of a cell to the size of the cytoplasm of that cell.The NC ratio may be expressed as a volumetric ratio or cross-sectionalarea. The NC ratio can indicate the maturity of a cell, with the size ofa cell nucleus decreasing with cell maturity. By contrast, high NC ratioin cells can be an indication of cell malignancy.

Signet ring morphology is the morphology of a signet ring cell, i.e., acell with a large vacuole, the malignant type of which appearspredominantly in cases of carcinoma. Signet ring cells are mostfrequently associated with stomach cancer, but can arise from any numberof tissues including the prostate, bladder, gallbladder, breast, colon,ovarian stroma and testis. Signet ring cell carcinoma (SRCC), forexample, is a rare form of highly malignant adenocarcinoma. It is anepithelial malignancy characterized by the histologic appearance ofsignet ring cells.

These biomarkers, TILS, NC ratio, ploidy, signet ring morphology, andPD-L1, are examples of biomarkers of morphological features determinedfrom labeled based images, in accordance with techniques herein.

Consensus molecular subtypes (“CMS”) are a set of classificationsubtypes of colorectal cancer (CRC) developed based on comprehensivegene expression profile analysis. CMS classifications in primarycolorectal cancer include: CMS1-immune infiltrated (Often BRAFmut,MSI-High, TMB-High); CMS2-canonical (Often ERBB/MYC/WNT driven);CMS3-metabolic (Often KRASmut); and CMS4-mesenchymal (Often TGF-Bdriven). More broadly, CMS herein includes these and other subtypes forcolorectal cancer. More broadly still, CMS herein refers subtypesderived from comprehensive gene expression profile analysis of othercancer types listed herein.

Homologous recombination deficiency (“HRD”) status is a classificationindicating deficiency in the normal homologous recombination DNA damagerepair process that results in a loss of duplication of chromosomalregions, termed genomic loss of heterozygosity (LOH).

Biomarkers such as CMS and HRD are examples of biomarkers ofmorphological features determined from labeled RNA data, in accordancewith techniques herein.

By way of example, biomarkers herein include HRD status, DNA ploidyscores, karyotypes, CMS scores, chromosomal instability (CIN) status,signet ring morphology scores, NC ratios, cellular pathway activationstatus, cell state, tumor characteristics, and splice variants.

As used herein, “histopathology images” refers to digital (includingdigitized) images of microscopic histopathology developed tissue.Examples include images of histological stained specimen tissue, wherehistological staining is a process undertaken in the preparation ofsample tissues to aid in microscopic study. In some examples, thehistopathology images are digital images of hematoxylin and eosin stain(H&E) stained histopathology slides, immunohistochemistry (IHC) stainedslides, Romanowsky Stains-Giemsa stained slides, Gram stained slides,Trichrome stained slides, carmine stained slides, and silver nitratestained slide. Other examples include blood smeared slides and tumorsmeared slides. In other examples, the histopathology images are ofother stained slides known in the art. As used herein, references todigital images, digitized images, slide images, and medical imagesrefers to “histopathology images.”

These histopathology images may be captured in visible wavelength regionas well as beyond the visible region, such as infrared digital imagesobtained using spectroscopic examination of histopathology developedtissue. In some examples, histopathology images include z-stack imagesthat represent horizontal cross-sections of a 3-dimensional specimen orhistopathology slide, captured at varying levels of the specimen orvarying focal points of the slide. In some examples, two or more imagesmay be from adjacent or near-adjacent sections of tissue from aspecimen, and one of the two or more images may have tissue featuresthat correspond with tissue features on another of the two or moreimages. There may be a vertical and/or horizontal shift between thelocation of the corresponding tissue features in the first image and thelocation of the corresponding tissue features in the second image. Thus,histopathology images also refers to images, sets of images, or videosgenerated from multiple different images. It should be understood thatthe following exemplary embodiments may be interchanged, or model'strained, with differing styles of staining unless explicitly excluded.

Various examples herein are described with reference to a particularclass of histopathology image, H&E slide images. A digital H&E slideimage may be generated by capturing a digital photograph of an H&Eslide. Alternatively or in addition, such an image may be generatedthrough machine learning systems, such as deep learning, from imagesderived from unstained tissue. For example, a digital H&E slide imagemay be generated from wide-field autofluorescence images of unlabelledtissue sections. See, e.g., Rivenson et al, Virtual histologicalstaining of unlabelled tissue-autofluorescence images via deep learning.Nature Biomedical Engineering, 3(6):466, 2019.

In various examples herein, the imaging-based biomarker predictionsystem may be implemented in whole or in part as Smart Pathology system,as described in reference to processes of FIGS. 39-41. As such,references herein to an imaging-based biomarker prediction systeminclude an implementation as an image-based nucleic acid yieldprediction system (or tumor cell prediction system), for example, asdescribed in various features and Aspects herein. For example, any ofthe systems of FIGS. 1, 3, 30, 38, and 40 may be implemented to performany of the features or Aspects described herein.

In an example, a Smart Pathology system may be configured to implement,in whole or in part, a computer-implemented method for qualifying aspecimen prepared on one or more hematoxylin and eosin (H&E) slides andfor providing associated unstained slides for nucleic acid analysis.That method may include generating a digital image of each of the one ormore H&E slides prepared from the specimen at an image-based nucleicacid yield prediction system having one or more processors. That methodmay further include, for each digital image, identifying, using the oneor more processors, tumor cells of the H&E slide from the digital image.The method may include generating a tumor area mask based at least inpart on the identified tumor cells, wherein the tumor area mask definesa dissection boundary of the H&E slide associated with the digitalimage. The method may include predicting an expected yield of nucleicacid for the tumor cells within the dissection boundary; and providingone or more of the associated unstained slides for nucleic acid analysiswhen the summation of expected yield of nucleic acid predicted for eachdigital image exceeds a predetermined threshold.

In an example, a Smart Pathology system may be configured to implement,in whole or in part, a computer-implemented method including: receivinga first hematoxylin and eosin (H&E) slide prepared from a tumor block atan image-based nucleic acid yield prediction system having one or moreprocessors. The method may further include identifying, using the one ormore processors, tumor cells of the H&E slide from a digital image. Themethod may further include generating a tumor area mask based at leastin part on the identified tumor cells, wherein the tumor area maskdefines a dissection boundary of the H&E slide associated with thedigital image. The method may further include predicting an expectedyield of nucleic acid for the tumor cells within the dissectionboundary. The method may further include based on the predicted expectedyield of nucleic acid, identifying a number of H&E slides to preparefrom the tumor block to satisfy a total nucleic yield; and preparing thenumber of H&E slides from the tumor block when the predicted expectedyield of nucleic acid exceeds a predetermined threshold.

In an example, a Smart Pathology system may be configured to implement,in whole or in part, a computer-implemented method for predicting anexpected yield of nucleic acid from tumor cells within a dissectionboundary on a hematoxylin and eosin (H&E) slide. The method may includereceiving a digital image of the H&E slide at an image-based nucleicacid yield prediction system having one or more processors. The methodmay include identifying, using the one or more processors, tumor cellsof the H&E slide from digital image using a trained cell segmentationmodel. The method may include generating a tumor area mask based atleast in part on the identified tumor cells, wherein the tumor area maskdefines the dissection boundary of the H&E slide. The method may includepredicting the expected yield of nucleic acid for the tumor cells withinthe dissection boundary.

FIG. 1 illustrates a prediction system 100 capable of analyzing digitalimages of histopathology slides of a tissue sample and determining thelikelihood of biomarker presence in that tissue, where biomarkerpresence indicates a predictive tumor presence, a predicted tumorstate/condition, or other information about a tumor of the tissuesample, such as a possibility of clinical response through the use of atreatment associated with the biomarker.

The system 100 includes an imaging-based biomarker prediction system 102that implements, among other things, image processing operations, deeplearning frameworks, and report generating operations to analyzehistopathology images of tissue samples and predict the presence ofbiomarkers in the tissue samples. In various examples, the system 100 isconfigured to predict the present of these biomarkers, tissuelocation(s) associated with these biomarkers, and/or cell location ofthese biomarkers. In various examples, the imaging-based biomarkerprediction system 102 may be implemented in whole or in part as SmartPathology system, as described in reference to processes of FIGS. 39-41and the Aspects described herein. As such, the imaging-based biomarkerprediction system may be implemented as an image-based nucleic acidyield prediction system (or tumor cell prediction system), for example,as described in various features and Aspects herein.

The imaging-based biomarker prediction system 102 may be implemented onone or more computing device, such as a computer, tablet or other mobilecomputing device, or server, such as a cloud server. The imaging-basedbiomarker prediction system 102 may include a number of processors,controllers or other electronic components for processing orfacilitating image capture, generation, or storage and image analysis,and deep learning tools for analysis of images, as described herein. Anexample computing device 3800 for implementing the imaging-basedbiomarker prediction system 102 is illustrated in FIG. 38.

As illustrated in FIG. 1, the imaging-based biomarker prediction system102 is connected to one or more medical data sources through a network104. The network 104 may be a public network such as the Internet,private network such as a research institution's or corporation'sprivate network, or any combination thereof. Networks can include, localarea network (LAN), wide area network (WAN), cellular, satellite, orother network infrastructure, whether wireless or wired. The network 104can be part of a cloud-based platform. The network 104 can utilizecommunications protocols, including packet-based and/or datagram-basedprotocols such as internet protocol (IP), transmission control protocol(TCP), user datagram protocol (UDP), or other types of protocols.Moreover, the network 104 can include a number of devices thatfacilitate network communications and/or form a hardware basis for thenetworks, such as switches, routers, gateways, access points (such as awireless access point as shown), firewalls, base stations, repeaters,backbone devices, etc.

Via the network 104, the imaging-based biomarker prediction system 102is communicatively coupled to receive medical images, for example ofhistopathology slides such as digital H&E stained slide images, IHCstained slide images, or digital images of any other staining protocolsfrom a variety of different sources. These sources may include aphysician clinical records systems 106 and a histopathology imagingsystem 108. Any number of medical image data sources could be accessibleusing the system 100. The histopathology images may be images capturedby any dedicated digital medical image scanners, e.g., any suitableoptical histopathology slide scanner including 20× and 40× resolutionmagnification scanners. Further still, the biomarker prediction system102 may receive images from histopathology image repositories 110. Inyet other examples, images may be received from a partner genomicsequencing system 112, e.g., the TCGA and NCI Genomic Data Commons.Further still, the biomarker prediction system 102 may receivehistopathology images from an organoid modeling lab 116. These imagesources may communicate image data, genomic data, patient data,treatment data, historical data, etc., in accordance with the techniquesand processes described herein. Each of the image sources may representmultiple image sources. Further, each of these image sources may beconsidered a different data source, those data sources may be capable ofgenerating and providing imaging data that differs from other providers,hospitals, etc. The imaging data between different sources potentiallydiffers in one or more ways, resulting in different data source-specificbias, such as in different dyes, biospecimen fixations, embeddings,staining protocols, and distinct pathology imaging instruments andsettings.

In the example of FIG. 1, the imaging-based biomarker prediction system102 includes an image pre-processing sub-system 114 that performsinitial image processing to enhance image data for faster processing intraining a machine learning framework and for performing biomarkerprediction using a trained deep learning framework. In the illustratedexample, the image pre-processing sub-system 114 performs anormalization process on received image data, including one or more ofcolor normalization 114 a, intensity normalization 114 b, and imagingsource normalization 114 c, to compensate for and correct fordifferences in the received image data. While in some examples theimaging-based biomarker prediction system 102 receives medical images,in other examples the sub-system 114 is able to generate medical images,either from received histopathology slides or from other receivedimages, such as generating composite histopathology images by aligningshifted histopathology images to compensate from vertical/horizontalshift. This image pre-processing allows a deep learning framework tomore efficiently analyze images across large data sets (e.g., over1000s, 10000s, to 100000s, to 1000000s of medical images), therebyresulting in faster training and faster analysis processing.

The image pre-processing sub-system 114 may perform further imageprocessing that removes artifacts and other noise from received imagesby doing preliminary tissue detection 114 d, for example, to identifyregions of the images corresponding to histopathology stained tissue forsubsequent analysis, classification, and segmentation.

As further described herein, in multiscale configuration where imagedata is to be analyzed on a tile-basis, in some examples, imagepre-processing includes receiving an initial histopathology image, at afirst image resolution, downsampling that image to a second imageresolution, and then performing a normalization on the downsampledhistopathology image, such as color and/or intensity normalization, andremoving non-tissue objects from the image.

In single-scale configurations, by contrast, downsampling of thereceived histopathology image is not used. Single-scale configurationsanalyze image data on a slide-level basis, not on a tile-basis.

In yet some hybrid versions of each of multiscale and single-scaleconfigurations a tiling process is imposed on received histopathologyimages to generate tiles for a tile-based analysis thereof.

The imaging-based biomarker prediction system 102 may be a standalonesystem interfacing with the external (i.e., third party)network-accessible systems 106, 108, 110, 112, and 116. In someexamples, the imaging-based biomarker prediction system 102 may beintegrated with one or more of these systems, including as part of adistributed cloud-based platform. For example, the system 102 may beintegrated with a histopathology imaging system, such as a digital H&Estain imaging system, e.g. to allow for expedited biomarker analysis andreporting at the imaging station. Indeed, any of the functions describedin the techniques herein may be distributed across one or more networkaccessible devices, including cloud-based devices.

In some examples, the imaging-based biomarker prediction system 102 ispart of a comprehensive biomarker prediction, patient diagnosis, andpatient treatment system. For example, the imaging-based biomarkerprediction system 102 may be coupled to communicate predicted biomarkerinformation, tumor prediction, and tumor state information to externalsystems, including a computer-based pathology lab/oncology system 118that may receive a generated biomarker report including image overlaymapping and use the same for further diagnosing cancer state of thepatient and for identifying matching therapies for use in treating thepatient. The imaging-based biomarker prediction system 102 may furthersend generated reports to a computer system 120 of the patient's primarycare provider and to a physician clinical records system 122 fordatabasing the patients report with previously generated reports on thepatient and/or with databases of generated reports on other patients foruse in future patient analyses, including deep learning analyses, suchas those described herein.

To analyze the received histopathology image data and other data, theimaging-based biomarker prediction system 102 includes a deep learningframework 150 that implements various machine learning techniques togenerate trained classifier models for image-based biomarker analysisfrom received training sets of image data or sets of image data andother patient information. With trained classifier models, the deeplearning framework 150 is further used to analyze and diagnose thepresence of image-based biomarkers in subsequent images collected frompatients. In this manner, images and other data of previously treatedand analyzed patients is utilized, through the trained models, toprovide analysis and diagnosis capabilities for future patients.

In the example system 100, the deep learning framework 150 includes ahistopathology image-based classifier training module 160 that canaccess received and stored data from the external systems 106, 108, 110,112, and 116, and any others, where that data may be parsed fromreceived data streams and databased into different data types. Thedifferent data types may be divided into image data 162 a which may beassociated with the other data types molecular data 162 b, demographicdata 162 c, and tumor response data 162 d. An association may be formedby labeling the image data 162 a with one or more of the different datatypes. By labeling the image data 162 a according to associations withthe other data types, the imaging-based biomarker prediction system maytrain an image classifier module to predict the one or more differentdata types from image data 162 a.

In the illustrated data, the deep learning framework 150 includes imagedata 162 a. For example, to train or use a multiscale PD-L1 biomarkerclassifier, this image data 162 a may include pre-processed image datareceived from the sub-system 114, images from H&E slides or images fromIHC slides (with or without human annotation), including IHC slidestargeting PD-L1, PTEN, EGFR, Beta catenin/catenin betal, NTRK, HRD,PIK3CA, and hormone receptors including HER2, AR, ER, and PR. To trainor use other biomarker classifiers, whether multiscale classifiers orsingle-scale classifiers, the image data 162A may include images fromother stained slides. Further, in the example of training a single scaleclassifier, the image data 162A is image data associated with RNAsequence data for particular biomarker clusters, to allow of multipleinstance learning (MIL) techniques herein.

The molecular data 162 b may include DNA sequences, RNA sequences,metabolomics data, proteomic/cytokine data, epigenomic data, organoiddata, raw karyotype data, transcription data, transcriptomics,metabolomics, microbiomics, and immunomics, identification of SNP, MNP,InDel, MSI, TMB, CNV Fusions, loss of heterozygosity, loss or gain offunction. Epigenomic data includes DNA methylation, histonemodification, or other factors which deactivate a gene or causealterations to gene function without altering the sequence ofnucleotides in the gene. Microbiomics includes data on viral infectionswhich may affect treatment and diagnosis of certain illnesses as well asthe bacteria present in the patient's gastrointestinal tract which mayaffect the efficacy of medicines ingested by the patient. Proteomic dataincludes protein composition, structure, and activity; when and whereproteins are expressed; rates of protein production, degradation, andsteady-state abundance; how proteins are modified, for example,post-translational modifications such as phosphorylation; the movementof proteins between subcellular compartments; the involvement ofproteins in metabolic pathways; how proteins interact with one another;or modifications to the protein after translation from the RNA such asphosphorylation, ubiquitination, methylation, acetylation,glycosylation, oxidation, or nitrosylation.

The deep learning framework 150 may further include demographic data 162c and tumor response data 162 d (including data about a reduction in thegrowth of the tumor after exposure to certain therapies, for exampleimmunotherapies, DNA damaging therapies like PARP inhibitors orplatinums, or HDAC inhibitors). The demographic data 162 c may includeage, gender, race, national origin, etc. The tumor response data 162 dmay include epigenomic data, examples of which include alterations inchromatin morphology and histone modifications.

The tumor response data 162 d may include cellular pathways, example ofwhich include IFNgamma, EGFR, MAP KINASE, mTOR, CYP, CIMP, and AKTpathways, as well as pathways downstream of HER2 and other hormonereceptors. The tumor response data 162 d may include cell stateindicators, examples of which include Collagen composition, appearance,or refractivity (for example, extracellular vs fibroblast, nodularfasciitis), density of stroma or other stromal characteristics (forexample, thickness of stroma, wet vs. dry) and/or angiogenesis orgeneral appearance of vasculature (including distribution of vasculaturein collagen/stroma, also described as epithelial-mesenchymal transitionor EMT). The tumor response data 162 d may include tumorcharacteristics, examples of which include the presence of tumor buddingor other morphological features/characteristics demonstrating tumorcomplexity, tumor size (including the bulky or light status of a tumor),aggressiveness of tumor (for example, known as high grade basaloidtumor, especially in colorectal cancer, or high grade dysplasia,especially in barrett's esophagus), and/or the immune state of a tumor(for example, inflamed/“hot” vs. non-inflamed/“cold” vs immuneexcluded).

The histopathology image-based classifier training module 160 may beconfigured with an image-analysis adapted machine learning techniques,including, for example, deep learning techniques, including, by way ofexample, a CNN model and, more particular, a tile-resolution CNN, thatin some examples is implemented as a FCN model, and, more particularlystill, implemented as a tile-resolution FCN model. Any of the data types162 a-162 d may be obtained directly from data communicated to theimaging-based biomarker prediction system 102, such as contained withinand communicated along with the histopathology images. The data types162 a-162 d may be used by the histopathology image-based classifiertraining module 160 to develop classifiers for identifying one of moreof the biomarkers discussed herein.

In one example, a histopathology image may be segmented and each segmentof the image may be labeled according to one or more data types that maybe classified to that segment. In another example, the histopathologyimage may be labeled as a whole according to the one or more data typesthat may be classified to the image or at least one segment of theimage. Data types may indicate one or more biomarkers and labeling ahistopathology image or a segment with a data type may identify thebiomarker.

In the example system 100, the deep learning framework 150 furtherincludes a trained image classifier module 170 that may also beconfigured with the deep learning techniques, including thoseimplementing the module 160. In some examples, the trained imageclassifier module 170 accesses the image data 162 for analysis andbiomarker classification. In some examples, the module 170 furtheraccesses the molecular data 162, the demographic data 162 c, and/ortumor response data 162 d for analysis and tumor prediction, matchedtherapy predictions, etc.

The trained image classifier module 170 includes trained tissueclassifiers 172, trained by the module 160 using one or more trainingimage sets, to identify and classify tissue type in regions/areas ofreceived image data. In some examples, these trained tissue classifiersare trained to identify biomarkers via the tissue classification, wherethese include single-scale configured classifiers 172 a and multiscaleclassifiers 172 b.

The module 170 may further include other trained classifiers, including,trained cell classifiers 174 that identify biomarkers via cellclassification. The module 170 may further include a cell segmenter 176that identifies cells within a histopathology image, including cellborders, interiors, and exteriors.

In examples herein, the tissue classifiers 172 may include biomarkerclassifiers specifically trained to identify tumor infiltration (such asby ratio of lymphocytes in tumor tissue to all cells in tumor tissue),PD-L1 (such as positive or negative status), ploidy (such as by ascore), CMS (such as to identify subtype), NC Ratio (such as nucleussize identification), signet ring morphology (such as a classificationof a signet cell or vacuole size), HRD (such as by a score, or by apositive or negative classification), etc. in accordance with thebiomarkers herein.

As detailed herein, the trained image classifier module 170 andassociated classifiers may be configured with an image-analysis adaptedmachine learning techniques, including, for example, deep learningtechniques, including, by way of example, a CNN model and, moreparticular, a tile-resolution CNN, that in some examples is implementedas a FCN model, and, more particularly still, implemented as atile-resolution FCN model, etc.

The system 102 further includes a tumor report generator 180 configuredto receive classification data from the trained tissue (biomarker)classifiers 172, the trained cell (biomarker) classifiers 174 and thecell segmenter 172 and determine tumor metrics for the image data andgenerate digital image and statistical data reports, where such outputdata may be provided to the pathology lab 118, primary care physiciansystem 120, genomic sequencing system 112, a tumor board, a tumor boardelectronic software system, or other external computer system fordisplay or consumption in further processes.

A conventional cancer diagnosis workflow 200 using histopathology imagesis shown in FIG. 2. A biopsy is performed to collect tissue samples froma patient. A medical lab generates digital histopathology images (202)for the tissue sample, for example using known staining techniques suchas H&E or IHC staining and a digital medical imager (for example, aslide scanner). These histopathology images are provided to apathologist who visually analyzes them (204) to identify tumors withinthe received image. The pathologist may optionally receive genomicsequencing data for the patient (e.g., DNA Seq data or RNA Seq data froma genomic sequencing lab) and analyze that data (206). From the visualanalysis of a histopathology slide and from the optional genomicsequencing data, the pathologist then diagnoses a cancer type othercharacteristics of the tumor/cancer cells (208) and generates apathology report (210).

FIG. 3 illustrates an example implementation of the imaging-basedbiomarker prediction system 102, and more particularly, of the deeplearning framework 150 in the form of deep learning framework 300. Theframework 300 may be communicatively coupled to receive histopathologyimage data and other data (molecular data, tumor response data,demographic data, etc.) from external systems, such as the physicianclinical records system 106, the histopathology imaging system 108, thegenomic sequencing system 112, the medical images repository 110, and/orthe organoid modeling lab 116 of FIG. 1 and through the network 104. Theorganoid modeling lab 116 may collect various types of data, such as,for example, the sensitivity of an organoid to a drug (for example,determined by measuring cell death or cell viability after exposure tothe drug), single-cell analysis data or detection of cellular products(including proteins, lipids, and other molecules) indicating thepresence of specific cell populations, including effector data,stimulatory data, regulatory data, inflammatory data, chemoattractivedata, as well as organoid image data, any of which may be stored withinthe molecular data 162 b.

The framework 300 includes a pre-processing controller 302, a deeplearning framework cell segmentation module 304, a deep learningframework multiscale classifier module 306, a deep learning frameworksingle-scale classifier module 307, and a deep learning post-processingcontroller 308.

To prepare the medical images for multiscale and single-scale deeplearning, in an example, the pre-processing controller 302 includesnormalization processes 310, which may include color normalization,intensity normalization, and imaging source normalization. Thenormalization process 310 is option and may be excluded to expedite deeplearning training, image analysis, and/or biomarker prediction.

An image discriminator 314 receives the normalized histopathology imagesfrom the normalization processes 310 and examines the images, includingimage metadata, to determine image type. The image discriminator 314 mayanalyze image data to determine if the image is a training image, e.g.,an image from a training dataset. The image discriminator 314 mayanalyze image data to determine the labeling type on the image, forexample, whether the image has a tile-level labeling, slide-levellabeling, or no labeling. The image discriminator 314 may analyze theimage data to determine the slide staining used to generate the digitalimage, H&E, IHC, etc.

In response to examining this image data, the image discriminator 314determines which images are to be provided to a slide-level labelpipeline 313 for feeding into the deep learning framework single-scaleclassifier module 307 and which images are to be provided to atile-level label pipeline 315 for feeding into the deep learningframework multiscale classifier 306.

In the illustrated example, images having tile-level labeling thepipeline 315 includes tissue detection processes and image tilingprocesses. These processes may be performed on all received imaged data,only on training image data, only on received image data for analysis,or some combination thereof. In some examples, the tissue detectionprocess, for example, may be excluded to expedite deep learningtraining, image analysis, and/or biomarker prediction. Indeed, any ofthe processes of the controller 302 may be performed in a dedicatedbiomarker prediction system or distributed for performance byexternally-connected systems. For example, a histopathology imagingsystem may be configured to perform normalization processes beforesending image data to the biomarker prediction system. In some examples,the biomarker prediction system may communicate an executablenormalization software package to the connected external systems thatconfigures those systems to perform normalization or otherpre-processing.

In examples in which the image discriminator 314 sends unlabeled imagesto the pipeline 315, the pipeline 315 includes a multiple instancelearning (MIL) controller, discussed further herein, configured toconvert all or portions of these histopathology images to tile-labeledimages. The MIL controller may be configured to perform processesherein, such as those described in FIGS. 18-26.

To expedite tissue detection of the trained tissue classifier, thetissue detection process of the pipeline 315 may perform initial tissueidentification, to locate and segment the tissue regions of interest forbiomarker analysis. Such issue tissue identification may include, forexample, identifying tissue boundaries and segmenting an image intotissue and non-tissue regions, so that metadata identifying the tissueregions is stored with the image data to expedite processing and preventbiomarker analysis attempts on non-tissue regions or on regions notcorresponding to the tissue to be examined.

To facilitate deep learning classification in various multiscaleconfigurations, the deep learning framework multiscale classifier module306 is configured to classify tissue using a tiling analysis. Forexample, in the pipeline 315, the tissue detection process sendshistopathology images (e.g., image data enhanced with tissue detectionmetadata) to the image tiling process that selects and applies a tilingmask to the received images to parse the images into small sub-imagesfor analysis by the framework module 306. The pipeline 315 may store aplurality of different tiling masks and select a tiling mask. In someexamples, the image tiling process selects one or more tiling masksoptimized for different biomarkers, i.e., in some examples, image tilingis biomarker specific. This allows, for example, to have tiles ofdifferent pixel sizes and different pixel shapes that are selectedspecifically to increase accuracy and/or to decrease processing timeassociated with a particular biomarker. For example, tile sizesoptimized for identifying the presence of TILs in an image may bedifferent from tile sizes optimized for identifying PD-L1 or anotherbiomarker. As such, in some examples, the pre-processor controller 302is configured to perform imaging processing and tiling specific to atype of biomarker, and after the system 300 analyzes image data for thatbiomarker, the controller 302 may re-process the original image data foranalyzing for the next biomarker, and so on, until all biomarkers havebeen examined for.

Generally speaking, the tiling masks applied by the image tiling processof the pipeline 315 may be selected to increase efficiency of operationof the deep learning framework module 306. The tiling mask may beselected based on the size of the received image data, based on theconfiguration of the deep learning framework 306, based on theconfiguration of the framework module 304, or some combination thereof.

Tiling masks may vary in the size of tiling blocks. Some tiling maskshave uniform tiling blocks, i.e., each the same size. Some tiling maskshaving tiling blocks of different sizes. The tiling mask applied by theimage tiling process may be chosen based on the number of classificationlayers in the deep learning framework 306, for example. In someexamples, the tiling mask may be chosen based on the processorconfiguration of the biomarker prediction system, for example, if themultiple parallel processors are available or if graphical processingunits or tensor processing units are used.

In the illustrated example, the deep learning multiscale classifiermodule 304 is configured to perform cell segmentation through a cellsegmentation model 316, where cell segmentation may be a pixel-levelprocess of the histopathology image from normalization process 310. Inother examples, this pixel-level process may be performed on image tilesreceived from the pipeline 315. In some examples, the cell segmentationprocess of the framework 304 results in classifications that biomarkerclassifications, because some of the biomarkers identified herein aredetermined from cell level analysis, in contrast to tissue levelanalysis. These include signet ring, large nuclei and high NC ratio, forexample. The module 304 may be configured using a CNN configuration, inparticular an FCN configuration for implementing each separatesegmentation.

The deep learning framework multiscale classifier module 306 includes atissue segmentation model 318, a tissue classification model 320, and abiomarker classification model 320. Like the module 304, the module 306may be configured using a CNN configuration, in particular an FCNconfiguration for implementing each separate segmentation.

In an example, the cell segmentation model 316 of the module 304 may beconfigured as a three-class semantic segmentation FCN model developed bymodifying a UNet classifier replacing a loss function with across-entropy function, focal loss function, or mean square errorfunction to form a three-class segmentation model. Three-class nature ofthe FCN model means that the cell segmentation model 316 may beconfigured as a first pixel-level FCN model, that identifies and assignseach pixel of image data into a cell-subunit class: (i) cell interior,(ii) a cell border, or (iii) a cell exterior. This is provided by way ofexample. The segmentation size of the module model 316 may be determinedbased on the type of cell to be segmented. For both TILs biomarkers, forexample, the model 316 may be configured to perform lymphocyteidentification and segmentation using a three-class FCN model. Forexample, the cell segmentation model 316 may be configured to classifypixels in an image as corresponding to the (i) interior, (ii) border, or(iii) exterior of lymphocyte cell. The cell segmentation model 316 maybe configured to identify and segment any number of cells, examples ofwhich include tumor positive, tumor negative, lymphocyte positive,lymphocyte negative, immune cells, including lymphocytes, cytotoxic Tcells, B cells, NK cells, macrophages, etc.

In some examples, the module 304 receives tiled, sub-images from thepipeline 315, and the cell segmentation model 316 determines the list oflocations of all lymphocytes, and those locations are compared to theother three class model's list of all cells determined from the model316 to eliminate any falsely detected lymphocytes that are not cells.The system 300 then takes this new list of locations of confirmedlymphocytes from the module 304 and compares to a tissue segmentermodule 318 list of tissue, e.g., tumor and non-tumor tissue locationsdetermined from tissue classification model 320 and determines whetherthe lymphocyte is in tumor or non-tumor region.

The use of a three-class model facilitates, among other things, thecounting of each individual cell, especially when two or more cellsoverlap each other for more accurate classification. Tumor infiltratinglymphocytes will overlap tumor cells. In traditional two-class celloutlining models that only label whether a pixel contains a cell outeredge or not, each clump of two or more overlapping cells would becounted as one cell, which can produce inaccurate results.

In addition to using a three-class model, the cell segmentation model316 may be configured to avoid the possibility that a cell that spanstwo tiles is counted twice, by adding a buffer around all four sides ofeach tile that is slightly wider than an average cell. The intention isto only count cells that appear in the center, non-buffered region foreach tile. In this case, tiles will be placed so that the center,non-buffered region of neighboring tiles are adjacent andnon-overlapping. Neighboring tiles will overlap in their respectivebuffer regions.

In one example, the cell segmentation algorithm of the model 316 may beformed of two UNet models. One UNet model may be trained with images ofmixed tissue classes, where a human analyst has highlighted the outeredge of each cell and classified each cell according to tissue class. Inone example, training data includes digital slide images where everypixel has been labeled as either the interior of a cell, the outer edgeof a cell, or the background which is exterior to every cell. In anotherexample, the training data includes digital slide images where everypixel has been labeled with a yes or no to indicate whether it depictsthe outer edge of a cell. This UNet model can recognize the outer edgesof many types of cells and may classify each cell according to cellshape or its location within a tissue class region assigned by thetissue classification module 320.

Another UNet model may be trained with images of many cells of a singletissue class, or images of a diverse set of cells where cells of onlyone tissue class are outlined in a binary mask. In one example, thetraining set is labeled by associating a first value with all pixelsshowing a cell type of interest and a second value to all other pixels.Visually, an image labeled in this way might appear as a black and whiteimage wherein all pixels showing a tissue class of interest would bewhite and all other pixels would be black, or vice versa. For example,the images may have only labeled lymphocytes. This UNet model canrecognize the outer edges of that particular cell type and assign alabel to cells of that type in the digital image of the slide.

Whereas, the cell segmentation model 316 is a trained cell segmentationmodel that may be used for cell detection, in some examples the model316 is configured as a biomarker detection model, configured as apixel-level classifier that classify pixels as corresponding to abiomarker.

Turning to the deep learning framework multiscale classifier module 306,the tissue segmentation model 318 may be configured in a similar mannerto the segmentation model 316, that is, as a three-class semanticsegmentation FCN model developed by modifying a UNet classifierreplacing a loss function with a cross-entropy function, focal lossfunction, or mean square error function to form a three-classsegmentation model. The model 318 may identify the interior, exterior,and boundary of various tissue types in tiles.

The tissue classification model 320 is a tile-based classifierconfigured to classify tiles as corresponding to one of a plurality ofdifferent tissue classifications. Examples of tissue classes include butare not limited to tumor, stroma, normal, lymphocyte, fat, muscle, bloodvessel, immune cluster, necrosis, hyperplasia/dysplasia, red bloodcells, and tissue classes or cell types that are positive (contain atarget molecule of an IHC stain, especially in a quantity larger than acertain threshold) or negative for an IHC stain target molecule (do notcontain that molecule or contain a quantity of that molecule lower thana certain threshold). Examples also include tumor positive, tumornegative, lymphocyte positive, and lymphocyte negative.

With the cell segmentation in a histopathology image generated by thecell segmentation model 316 and the tissue classification from thetissue classification model 302, a biomarker classification model 322receives data from both and determines a predicted biomarker presence inthe histopathology image, and in particularly, with the multiscaleconfiguration, the prediction biomarker presence in each tile image ofthe histopathology image. The biomarker classification model 322 may bea trained classifier implemented in the deep learning framework model306 as shown or implemented separate from the model 306, such as in thedeep learning post-processing controller 308.

In some examples of the biomarker classification model detecting a TILSbiomarker, the tissue classification model 320 is trained to identify apercentage TILs within a tile image, the cell segmenter 316 determinesthe cell boundary, and the biomarker classification model 322 classifiesthe tile image based on the percentage of TILs within a cell interiorresulting in a classification: (i) Tumor—IHC/Lymphocyte positive or (ii)Non-tumor—IHC/Lymphocyte positive.

In some examples of the biomarker classification model detecting aploidy, the biomarker classification model 322 may be trained withploidy model based off of histopathology images and associated ploidyscores using techniques provided, for example, in Coudray N, Ocampo P S,Sakellaropoulos T, Narula N, Snuderl M, Fenyo D, et al., Classificationand mutation prediction from non-small cell lung cancer histopathologyimages using deep learning. Nat Med. 2018; 24:1559-67.)

In an example, the training data may be data such as real karyotypes,determined by cytogeneticists, although in some examples, the biomarkerclassification model 322 may be configured to infer such data. Theploidy data may be formatted in columns of: chromosome number, startposition, stop position, region length. Ploidy scores may be determinedfrom DNA sequencing data and may be specific to a gene, chromosome, oran arm of a chromosome. The ploidy score may be global and mightdescribe the entire genome of the sample (global CNV/copy numbervariation may cause a change in hematoxylin staining of the tumornuclei), and may be a score calculated by averaging the ploidy score foreach region in the genome, where a local, regional ploidy score might beweighted according to the length of each region that is associated withthat score. The trained ploidy model of the biomarker classificationmodel 322 may be specific to a gene, an arm of a chromosome, or anentire chromosome, because each section might affect the cell morphologyseen in histopathology images differently. Predicted ploidy biomarkerdata may affect accept/reject analysis, because if the tumor purity orcell count on a slide is low but the ploidy is higher than usual, theremight still be enough material for genetic testing. A biomarker metricsprocessor 326 may be configured to make such determinations prior toreport generation.

In some examples of the biomarker classification model detecting asignet ring morphology, the biomarker classification model 322 may betrained with a signet ring morphology model based off classificationtechniques such as the poorly cohesive (PC), signet ring cell (SRC), andLauren sub-classifications and others described in Mariette, C.,Carneiro, F., Grabsch, H. I. et al. Consensus on the pathologicaldefinition and classification of poorly cohesive gastric carcinoma.Gastric Cancer 22, 1-9 (2019) and other signet ring morphologyclassifications.

In some examples of the biomarker classification model detecting a NCRatio, the cell segmentation model 316 may be configured withthree-class UNet described herein, but where the model is trained toidentify three classes: nucleus, cytoplasm, and cell border/non-cellbackground. In an example, the training data may be images where eachpixel is manually annotated with one of these three classes, and/orimages that have been annotated in this way by a trained model, asdiscussed in the example of FIG. 4, updated training images.

Thus, the cell segmentation model 316 may be trained to analyze an inputimage and assign one of the three classes to each pixel, define cells asa group of adjacent nucleus pixels and all cytoplasm pixels between thenucleus pixels and the next nearest border pixels, and then for eachcell, the biomarker classification model 322 may be configured tocalculate the nucleus:cytoplasm ratio as the area (number of pixels) ofthe cell's nucleus divided by the area (number of pixels) of the entirecell (nucleus and cytoplasm).

To identify tumor tissue and tumor status for tissue, the deep learningframework 306 may be configured using a FCN classifier in an example.Whereas in an example the deep learning framework 304 may be configuredas a pixel-resolution FCN classifier, the deep learning framework 306may configured as a tile-resolution FCN classification model, ortile-resolution CNN model, i.e., performing a classification for anentire received tile of image data.

The classification model 320 of the module 306, for example, may beconfigured to classify tissue in a tile as corresponding to one of anumber of tissue classes, such as biomarker status, tumor status, tissuetype, and/or tumor state/condition, or other information. The module306, in the illustrated example, is configured having a tissueclassification 320 and a tissue segmentation model 322. In an exampleimplementation of a TILs biomarker, the tissue classification model 320may classify tissue using tissue classifications, such as Tumor—IHCpositive, Tumor—IHC negative, Necrosis, Stroma, Epithelium, or Blood.The tissue segmentation model 328 identifies boundaries for thedifferent tissue types identified by the tissue classification model 320and generates metadata for use in visually display boundaries and colorcoding for different tissue types in an overlay mapping report generatorby the post-processing controller 308.

In an example implementation, the deep learning framework 300 performsclassifications on a tile-basis by receiving tiles (i.e., sub-images)from the processes of classification models 320 and 322. In someexamples, tiling may be performed by the framework 306 using a tilingmask, and in addition to performing tissue classifications the module306 itself may send the generated sub-images to the module 304 forpixel-level segmentation. The module 306 may examine each tile in asequential manner, one after another, or the module 306 may examine eachtile in parallel by the nature of the matrix generated by the FCN modelfor faster processing of images.

In some examples, the tissue segmentation model 318 receivespixel-resolution cell segmentation data and/or pixel-resolutionbiomarker segmentation data from the module 304 and performs statisticalanalysis on a tile basis and on an image basis. In some examples, thatstatistical analysis determines (i) the area of image data covered bytissue, e.g., the area of the stained histopathology slide covered bytissue, and (ii) the number of cells in the image data, e.g., the numberof cells in the stained histopathology slide. The tissue segmentationmodel 318, for example, can accumulate cell and tissue classificationsfor each tile of an image until all tiles forming the image have beenclassified.

Where the deep learning framework is a multiscale classifier module forclassifying biomarkers on tile-basis, the deep learning framework 300 isfurther configured to classify biomarkers using classifications trainedfrom a slide-level training images, without the need for tile-levellabeling. For example, and as further discussed hereinbelow, slide-leveltraining images received by the image discriminator may be provided tothe slide level label piper 313 having a MIL controller configured toperform processes herein, such as those described in FIGS. 18-26, togenerate a plurality of tile images with inferred classifications, andoptionally perform tile selection on those tiles to train a tissueclassification model 317 and a biomarker classification model 319.Example single-scale classifiers include a CMS biomarker classificationmodel, where the output is a CMS class, and an HRD biomarkerclassification model, where the output is HRD+ or HRD−. Theseclassifications may be performed for an entire histopathology image todetermine a biomarker prediction or on each tile image of that digitalimage and analyzed by the biomarker metrics processor 326 to determine abiomarker prediction from the tile images.

In some examples of the biomarker classification model detecting HRD,the biomarker classification model 319 may be configured to predict HRD.Training of a HRD model within the classifier 318 may be based off ofhistopathology images and matched HRD score. For example, training datamay be generated by H&E images and RNA sequence data, including, in someexamples, RNA expression profile data that is fed to an HRD model andfed back in for further training, as with the updated training data 403in Fi. 4. The training data may be derived from tumor organoids: H&Eimages from an organoid paired with a measure of the organoid'ssensitivity to PARP inhibitors indicating HRD or a result of the RNAteam's HRD model run on the organoid's RNA expression profile. Anexample HRD prediction model of the biomarker classification model 319is described in Peng, Guang et al. Genome-wide transcriptome profilingof homologous recombination DNA repair, Nature communications vol. 5(2014): 3361 and van Laar, R. K., Ma, X.-J., de Jong, D., Wehkamp, D.,Floore, A. N., Warmoes, M. O., Simon, I., Wang, W., Erlander, M., van′tVeer, L. J. and Glas, A. M. (2009), Implementation of a novelmicroarray-based diagnostic test for cancer of unknown primary. Int. J.Cancer, 125: 1390-1397.

In an example, a deep learning framework may identify HRD from an H&Eslide using RNA expression to identify a slide level label indicative ofthe percentage of the slide that contains a biomarker expressing cell.In one example, an activation map approach for the RNA label may beapplied to the whole slide, either as a binary label (i.e. Positive orNegative HRD expression somewhere in the tissue), or as a continuouspercentage (i.e. 62% of cells in the image were found to express HRD). Abinary RNA label may be generated by next generation sequencing of aspecimen and a cell percentage label may be generated by applying singlecell RNA sequencing. In one example, single cell sequencing may identifycell-types and quantities present in the RNA expression from the NGS.

Training a tile based deep learning network to predict a biomarkerclassification label for each tile of the whole slide image may beperformed using any of the methods described herein. Once trained, themodel may be applied to a method of activation mapping to each tile.Activation Mapping may be performed using Grad-CAM (gradient classactivation mapping) or guided back-propagation. Both allowidentification of which regions of a tile contribute most to theclassification. In one example, the parts of the tile that contributemost to the HRD positive class may be cells clustered in the upper rightcorner of a tile. Cells within the identified active regions may then belabeled HRD positive cells.

Proving that the model performs to clinical certainty may includecomparing model results to a source of ground truth. One possiblegeneration of ground truth may include isolating small regions oftissue, each containing <100 cells by segmenting through a tissuemicroarray and sequencing each region separately to get RNA labels ofeach region. Generating a ground truth may further include classifyingthese regions with a biomarker classification model, and identifyingwith what accuracy the activation maps highlight cells in regions thathave high HRD expression and ignore most cells in regions that have lowHRD expression.

When training a tile based deep learning network to predict a biomarkerclassification label for each tile utilizes a strongly supervisedapproach to generate biomarker labels to identify the HRD status(Positive or Negative) of individual cells. Single cell RNA sequencingmay be used alone, or in combination with laser guided micro-dissectionto extract one cell at a time, to achieve labels for each cell. In oneexample a cell segmentation model may be incorporated to first get theoutline of the cells, then an artificial intelligence engine mayclassify the pixel values inside each of the cell contours according tobiomarker status. In another example masks of the image may be generatedwhere HRD positive cells are assigned a first value and HRD negativecells are assigned a second value. A single scale deep learningframework may then be trained using slides with masks to identify cellsthat express HRD.

In some examples of the biomarker classification model detecting CMS,the biomarker classification model 319 may be configured to predict CMS.Such biomarker classification may be configured to classify segmentedcells as corresponding to cancer specific classifications. For example,four trained CMS classifications in primary colorectal cancer include:1-immune infiltrated (Often BRAFmut, MSI-High, TMB-High); 2-canonical(Often ERBB/MYC/WNT driven); 3-metabolic (Often KRASmut); and4-mesenchymal (Often TGF-B driven). In other examples more trained CMSclassifications may be used, but generally two or more CMS subtypes areclassified in examples herein. Further still, other cancer types mayhave their own trained CMS categories, and the classifier 318 may beconfigured to have a model for subtyping each cancer type. Exampletechniques for developing CMS classifications of 4, 5, 6, 7 or greaternumbers of CMS classifications are using the CMSCcaller described inEide, P. W., Bruun, J., Lothe, R. A. et al. CMScaller: an R package forconsensus molecular subtyping of colorectal cancer pre-clinical models.Sci Rep 7, 16618 (2017) and https://github.com/peterawe/CMScaller.

Training of a CMS model within the classifier 318 may be based off ofhistopathology images matched CMS category assignment. CMS categoryassignment may be based on RNA expression profiles and in one example,are generated by an R program that uses Nearest Template Prediction,called CMS Caller (see, Eide, P. W., Bruun, J., Lothe, R. A. et al.CMScaller: an R package for consensus molecular subtyping of colorectalcancer pre-clinical models. Sci Rep 7, 16618 (2017) andhttps://github.com/peterawe/CMScaller). Alternative classificationsusing random forest model are described in Guinney, J., Dienstmann, R.,Wang, X. et al. The consensus molecular subtypes of colorectal cancer.Nat Med 21, 1350-1356 (2015). For example, a CMS Caller looks at eachRNA sequence data sample to determine is each gene over or under themean and gives a binary classification for each gene. This avoids batcheffects, for example between different RNA sequence data sets. Trainingdata could also include DNA data, IHC data, mucin markers, therapyresponse/survival data from clinical reports. These may or may not beassociated with a CMS category assignment. For example, CMS 4 IHC shouldstain positive for TGFbeta, CMS 1 IHC should be positive for CD3/CD8,CMS 2 and 3 have mucin gene changes, CMS 2 responds to cetuximab, CMS 1responds better to avastin. CMS 1 has best survival prognosis, CMS 4 hasworst. (see slide 12 of CMS slides). CMS categories 1 and 4 can bedetected from H&E. With training, the architecture of FIG. 4, forexample, can be used to train the model to identify and classify thedifference between CMS 2 and 3.

In an example, the biomarker classification model 319 may be configuredto identify five CRC intrinsic subtypes (CRIS) endowed with distinctivemolecular, functional and phenotypic peculiarities: (i) CRIS-A:mucinous, glycolytic, enriched for microsatellite instability or KRASmutations; (ii) CRIS-B: TGF-β pathway activity, epithelial-mesenchymaltransition, poor prognosis; (iii) CRIS-C: elevated EGFR signalling,sensitivity to EGFR inhibitors; (iv) CRIS-D: WNT activation, IGF2 geneoverexpression and amplification; and (v) CRIS-E: Paneth cell-likephenotype, TP53 mutations. CRIS subtypes successfully categorizeindependent sets of primary and metastatic CRCs, with limited overlap onexisting transcriptional classes and unprecedented predictive andprognostic performances. See, e.g., Isella, C., Brundu, F., Bellomo, S.et al. Selective analysis of cancer-cell intrinsic transcriptionaltraits defines novel clinically relevant subtypes of colorectal cancer.Nat Commun 8, 15107 (2017).

For biomarker detection, the biomarker classification model 319 may betrained with a CMS model that predicts for each tile, a CMSclassification, identifying different tissue types (e.g., stroma) forthat classification, in place of trying to mere average CMSclassification across all tiles. In an example, each tile would beprocessed, and the CMS model would generate a compressed representationwith each tile's associated pixel data and each tile would be assignedinto a class (cluster 1, cluster 2, etc.), based on patterns in eachtile's pixel data and similarities among the tiles. The list of thepercentage of tiles in the image that belong to each cluster could be acluster profile for the image and provided by a report generator. In anexample, each profile would be fed to the model with a corresponding CMSdesignation or RNA expression profile (which was the original methodused to define CMS categories) for training. In another example, eachtile from every training slide mage is annotated according to theoverall CMS category assigned to the entire slide from which the tileoriginated, then the tiles are clustered and analyzed to determine whichclusters are most closely associated with CMS category.

In some examples, the biomarker classification model 319 (as well sa thebiomarker classification model 322) may cluster each tile into adiscreet number of clusters, in place of performing the sameclassification on each tile and weighting that tile classificationequally. One way to achieve is to include attention layer in thebiomarker classification model. In an example, all tiles from alltraining slides may be categorized into a duster, then, if the number oftiles in a duster isn't statistically related to biomarker, that dusteris not weighted as high as a cluster that is associated with biomarker.In other examples, majority voting techniques may be used to train thebiomarker classification model 319 (or model 322).

While shown as separate models, the biomarker classification models 322and 319 may each be configured to include all or parts of correspondingtissue classification models, cell segmentation models, and tissuesegmentations models, as may be the case for classifying variousbiomarkers herein. Furthermore, while the biomarker classification model322 is shown contained within the multiscale classifier module 306 andthe biomarker classification model 319 is shown contained within thesingle-scale classifier module 307, in some examples, all or parts ofthese biomarker classification models may be implemented in thepost-processing controller 308, as may be the case for classifyingvarious biomarkers herein. Further, while described as tile-level orslide-level classification models, in some examples, the biomarkerclassification models 322 and 319 may be configured as pixel-levelclassifiers, in some examples.

From the determinations made by the modules 304, 306, and 307, thepost-processing controller 308 can determine whether the image datacontains an amount of tissue that exceeds a threshold and/or satisfies acriterion, for example enough tissue for genetic analysis, enough tissueto use the image data as training images during a learning phase of thedeep learning framework, or enough tissue to combine the image data withan already existing trained classifier model.

Thus, in various examples herein, including those described in referenceto FIG. 3 and elsewhere herein, a patient report may be generated. Thereport may be presented to a patient, physician, medical personnel, orresearcher in a digital copy (for example, a JSON object, a pdf file, oran image on a website or portal), a hard copy (for example, printed onpaper or another tangible medium), as audio (for example, recorded orstreaming), or in another format.

The report may include information related to gene expression calls (forexample, overexpression or underexpression of a given gene), detectedgenetic variants, other characteristics of a patient's sample and/orclinical records. The report may further include clinical trials forwhich the patient is eligible, therapies that may match the patientand/or adverse effects predicted if the patient receives a giventherapy, based on the detected genetic variants, other characteristicsof the sample and/or clinical records.

The results included in the report and/or additional results (forexample, from the bioinformatics pipeline) may be used to analyze adatabase of clinical data, especially to determine whether there is atrend showing that a therapy slowed cancer progression in other patientshaving the same or similar results as the specimen. The results may alsobe used to design tumor organoid experiments. For example, an organoidmay be genetically engineered to have the same characteristics as thespecimen and may be observed after exposure to a therapy to determinewhether the therapy can reduce the growth rate of the organoid, and thusmay be likely to reduce the growth rate of the patient associated withthe specimen.

In an example, the post-processing controller 308 is further configuredto determine a number of different biomarker prediction metrics and anumber of tumor prediction metrics, e.g., using a biomarker metricsprocessing module 326. Example prediction metrics include: tumor purity,number of tiles classified as a particular tissue class, number ofcells, number of tumor infiltrating lymphocytes, clustering of celltypes or tissue classes, densities of cell types or tissue classes,tumor cell characteristics—roundness, length, nuclei density, stromathickness around tumor tissue, image pixel data stats, predicted patientsurvival, PD-L1 status, MSI, TMB, origin of a tumor, andimmunotherapy/therapy response.

For example, the biomarker metrics processing module 326 may determinethe number of tiles classified in one or more single tissue classes, thepercentage of tiles classified in each tissue class, the ratio of thenumber of tiles classified in a first tissue class versus the numberclassified in a second tissue class for any two classes, and/or thetotal area of tiles classified in a single tissue class, for each tissueclass. The module 326 may determine tumor purity, based on number oftiles classified as tumor versus other tissue classes, or based onnumber of cells located in tumor tiles versus number of cells located inother tissue class tiles. The module 326 may determine the number ofcells for the entire histopathology image, within an area pre-defined bya user, within tiles classified as any one of the tissue classes, withina single grid tile, or over some area or region of interest, whetherpredetermined, selected by a user during operation of the system 300, orautomatically by the system 300, for example, by determining amost-likely region of interest based on the image analysis. The module326 may determine the clustering of cell types of tissue classes basedon spacing and density of classified cells, spacing and distance oftissue class classified tiles, or any visually detectable features. Insome examples, the module 326 determines the probability that twoneighboring cells will be either two immune cells, two tumor cells, orone of each, for example. The module 326 determines tumor cellcharacteristics by determining average roundness, perimeter length,and/or nuclei density of identified tumor cells. The thickness ofidentified stroma may be used as a predictor of patient response totreatment. The image pixel data stats determined by the module 326 mayinclude the mean, standard deviation, and sum for each tile of either asingle image or of an aggregate of images for any pixel data, includingthe following: red green blue (RGB) value, optical density, hue,saturation, grayscale, and stain deconvolution. Further, the module 326may calculate the location of lines, patterns of alternating brightness,outlines of shapes, staining patterns for segmented tissue classesand/or for segmented cells in the image. In any of these examples, themodule 326 may be configured to make the determination/predicted status,and an overlay display generation module 324 then generates a report fordisplaying the determined information. For example, the overlay mapgeneration module 324 may generate a network accessible user interfacethat allows a user to select different types of data to be displayed,and the module 324 generates an overlay map showing the selecteddifferent types of data overlaid on a rendition of the original stainedimage data.

FIG. 4 illustrates a machine learning data input/flow schematic 400 thatmay be implemented with the system 300 of FIG. 3 or, more generally,with any of the systems and processes described herein.

In training mode, where the deep learning frameworks in the system 300are trained, various training data may be obtained. In the illustratedexample, training image data 401 in the form of high resolution and lowresolution histopathology images are provided to the pre-processingcontroller 302. As shown, the training images may include annotatedtissue image data from various tissue types, e.g., tumor, stroma,normal, immune cluster, necrosis, hyperplasia/dysplasia, and red bloodcells. As shown, the training images may include computer generated,synthetic image data, as well as image data of segmented cells (cellimage data) and image data of labeled biomarkers (e.g., the biomarkersdiscussed herein), either slide-level label or tile-level labels(collectively, biomarker labeled image data). These training images maybe digitally annotated, but in some examples, the tissue annotations aremanually done. In some images, the training image data includesmolecular data and/or demographic data, for example, as metadata withinthe image data. In the illustrated example, such data is separately fedto a deep learning framework 402 (formed of example implementations of amultiscale deep learning framework 306′ and a single-scale deep learningframework 307′). Other training data may also be provided to thecontroller 302, such as pathway activation scores, for additionaltraining of the deep learning framework.

In some examples, the deep learning framework 402 generates updatedtraining images 403 that are annotated and segmented by the deeplearning framework 402 and fed back into the framework 402 (orpre-processing controller 302) for use in updated training of theframework 402.

In a diagnostic mode, patient image data 405 is provided to thecontroller 302, for use in accordance with the examples herein.

Any of the image data herein, including patient image data and trainingimage, may be histopathology image data, such as H&E slide images and/orIHC slide images. For IHC training images, for example, the images maybe segmented images that differentiate between cytotoxic and regulatoryt cells, or other cell types.

In some examples, the controller 302 generates image tiles 407,accessing one or more tiling masks 409, and tile metadata 411, which thecontroller 302 feeds as inputs to the deep learning framework 402, fordetermining predicted biomarker and/or tumor status and metrics, whichare then provided to an overlay report generator 404 for generating abiomarker and tumor report 406. Optionally, the report 406 may includean overlay of the histopathology image and further include biomarkerscoring data, such as percentage TILs, in an example.

In some examples, clinical data 413 is provided to the deep learningframework 402 for use in analyzing image data. Clinical data 413 mayinclude health records, biopsy tissue type, anatomical location ofbiopsy. In some examples, tumor response data 415 collected from apatient after therapy is additionally provided to the deep learningframework 402 for determining changes in biomarker status, tumor status,and/or metrics thereof.

FIG. 5 illustrates an example deep learning framework 500 formed of aplurality of different biomarker classification models. Elements in FIG.5 are provided as follows. “Cell” refers to a cell segmentation model,e.g., a trained pixel-level segmentation model, in accordance with theexamples herein. “Multi” refers to a multiscale (tile-based) tissueclassification model, in accordance with the examples herein. “Post”refers to arithmetic calculations that may be performed in the finalstages of a biomarker classification model configured to predictbiomarker status in an image or in a tile image in response to one ormore data from the “Cell” or “Multi” stages, in accordance with theexamples herein. In one example, a “Post” may include identifying amajority vote such as summing the number of tiles in an image associatedwith each biomarker label and assigning the biomarker status in theimage to the biomarker label with the largest sum. A two layer “Post”configuration refers to a two stage post-processing configuration wherearithmetic calculations may be stacked. In one example, a first layer ofpost may include summing cells within a tile labeled tissue and summinglymphocyte cells within the same tile. A second layer may divide thelymphocyte cell count by the cell count to generate a ratio, which whencompared to a threshold may be used to assign the biomarker status inthe image based on whether the ratio exceeds the threshold. The final“Post” configuration may include other post-processing functionalitysuch as report generation processes described herein. “Single” refers toa single-scale classification model, in accordance with the examplesherein. “MIL” refers to an MIL controller, in accordance with examplesherein. In the illustrated example, the deep learning framework 500contains a TILS biomarker classification model 502, a PD-L1classification model 504, a first CMS classification model 506 based ona “Single” classification architecture, a second CMS classificationmodel 508 based on a “Multi” classification architecture, and an HRDclassification model 510. Patient data 512, such as molecular data,demographic data, tumor response data, and patient images 514 are storedin datasets accessible by the deep learning framework 500.

Training data is also shown in the form of cell segmentation trainingdata 516, single-scale classification biomarker training data 518,multiscale classification biomarker training data 520, MIL training data522, and post-processing training data 524.

FIG. 6 illustrates a process 600 that may be executed by theimaging-based biomarker prediction system 102, the deep learningframework 300, or the deep learning framework 402, in particular in adeep learning framework having a multiscale configuration.

As part of a training process, at a block 602, tile-labeledhistopathology images are received at the deep learning framework 300.The histopathology images may be of any type herein, but are illustratedas digital H&E slide images in this example. These images may betraining images of a previously-determined and labeled (and thus known)cancer type (e.g., for supervised learning configurations). In someexamples, the images may be training images of a plurality of differentcancer types. In some examples, the images may be training images thatcontain some or all images of an unknown or unlabeled cancer type (e.g.,for unsupervised learning configurations). In some examples, thetraining images include digital H&E slide images with tissue classesannotated (for training the tile-resolution FCN tissue classifier) andother, digital H&E slide images with each cell annotated. In the exampleof training of TILS biomarker classification, each lymphocyte may beannotated in the H&E slide images (e.g., for training thepixel-resolution FCN segmentation classifiers annotated images to trainUNet model classifiers). In some examples, the training images may bedigital IHC stained images to train the pixel-resolution FCNsegmentation classifiers, in particular images where IHC stainingtargets lymphocyte markers. In some examples, the training images willinclude molecular data, clinical data, or images paired with otherannotations (like pathway activation scores, etc.).

In the illustrated example, at a block 604, a pre-processing isperformed on the training images, such as the normalization processesdescribed herein. Other pre-processing processes described herein mayalso be performed at the block 604.

At block 606, the tile-labeled H&E slide training images are provided toa deep learning framework and analyzed within the machine learningconfiguration thereof, such as a CNN and, more particular, atile-resolution CNN, in some examples implemented as a FCN model, and,for analyzing tile images of the training images for tissueclassification training, pixels of the training images for cellsegmentation training, and in some examples tile images for biomarkerclassification training. The result, a block 608 generates a traineddeep learning framework multiscale biomarker classification model, whichmay include a cell segmentation model and a tissue classification model.In training multiple biomarker classification models, the block 608 maygenerate a separate model for each of the biomarkers TILS, PD-L1,ploidy, NC ratio, and signet ring morphology.

As a prediction process, at a block 610 a new unlabeled histopathologyimage, such as a H&E slide image, is received and provided to themultiscale biomarker classification model, which at a block 612 predictsbiomarker status for the received histopathology image as determined bythe one or more biomarker classification models.

For example, new (unlabeled or labeled) histopathology images may bereceived at the block 610 from the physical clinical records system orprimary care system and applied to a trained deep learning frameworkwhich applies its trained cell segmentation, tissue classificationmodel, and biomarker classification models, and the block 612 determinesa biomarker prediction score. That prediction score can be determinedfor an entire histopathology image or for different regions across theimage. For example, for each image, the block 612 may generate anabsolute count of how many biomarkers are on the image, a percentage ofthe number of cells in tumor regions that are associated with each ofthe biomarkers, and/or a designation of any biomarker classifications orother information. In some examples, the deep learning framework mayidentify predicted biomarkers for all identified tissue classes withinthe image. As such, a biomarker prediction probability score may varyacross the image. For example, in predicting the presence of TILs, theprocess 612 may predict the presence of TILs at different locationswithin a histopathology image. TILs prediction would vary across theimage, as a result. This is provided by way of example, block 612 maydetermine any number of metrics discussed herein.

As shown in process 900 of FIG. 9, after prediction, the predictedbiomarker classification may be received at report generator a block902. A clinical report for the histopathology image, and thus for thepatient, may be generated at a block 904 including predicted biomarkerstatus and, at a block 906, an overlay map may be generated showing thepredicted biomarker status for display to a clinician or for providingto a pathologist for determining a preferred immunotherapy correspondingto the predicted biomarker.

In FIG. 7, an example process 700 for determining predicted biomarkerstatus, in particular predicting TILs status, is provided. Although theprocess 700 may be used to predict the status of any number ofbiomarkers and other metrics in accordance with the examples describedherein.

A pre-processing controller receives histopathology images (702) andperforms initial image processing, as described herein. In an example,the deep learning pre-processing controller receives an entire imagefile in any pyramidal TIFF format and identifies edges and outlines(e.g., performs a segmentation) of the viable tissue in the image. Theoutput of block 702 may be a binary mask of the input image, e.g., witheach pixel having a 0 or 1 value, where 0 denotes background and 1denotes foreground/tissue. The dimensions of the mask may be thedimensions of the input slide when downsampled 128×. This binary maskmay be temporarily buffered and provided to a tiling process 704.

At the process 704, the pre-processing controller applies a tissue maskprocess using a tiling procedure to divide the image into sub-images(i.e., tiles) that will be examined individually. Since the deeplearning framework is configured to perform two different learningmodels (one for tissue classification and one for cell/lymphocytesegmentation), different tiling procedures for each model may beperformed by the procedure 704. The process 704 may generate twooutputs, each output containing a list of coordinates, e.g., definedfrom the upper-left most corner of tiles. The output lists may betemporarily buffered and passed to tissue classification and cellsegmentation processes.

In the example of FIG. 7, tissue classification is performed at aprocess 706, which receives the histopathology image from the process704 and performs tissue classification on each received tile using atrained tissue classification model. The trained tissue classificationmodel is configured to classify each tile into different tissue classes(e.g., tumor, stroma, normal epithelium, etc.). Multiple layers oftiling may be used by the process 706 to reduce computationalredundancy. For each tile, the trained tissue classification modelcalculates the class probability for each class stored in the model. Theprocess 706 then determines the most likely class and assigns that classto the tile. The process 706 may output, as a result, a list of lists.Each nested interior list serves as nested classification that describesa single tile and contains the position of the tile, the probabilitiesthat the tile is each of the classes contained in the model, and theidentity of the most probable class. This information is listed for eachtile. The list of lists may be saved into the deep learning frameworkpipeline output json file.

At processes 708 and 710, cell segmentation and lymphocyte segmentationare performed, respectively. The processes 708 and 710 receive thehistopathology image and the tile list from processes 704 and 706. Theprocess 708 applies the trained cell segmentation model; and the process710 applies the trained lymphocyte segmentation model. That is, in theillustrated example, for each tile in the cell segmentation tile list,two pixel-resolution models are run in parallel. In an example, the twomodels both use a UNet architecture but have been trained with differenttraining data. The cell segmentation model identifies cells and draws aborder around every cell in the received tile. The lymphocytesegmentation model identifies lymphocyte and draws a border around everylymphocyte in the tile. Because Hematoxylin binds to DNA, performing“cell segmentation” using digital H&E slide images may also be referredto as nuclei segmentation. That is, the cell segmentation model process708 performs nuclei segmentation on all cells, while the lymphocytesegmentation model process 710 performs nuclei segmentation onlymphocytes.

Because, in this example, the same UNet architecture is used for both,the processes 708 and 710 each produce two identically-formatted maskarray outputs. Each output is a mask array with the same shape and sizeas the received tile. Each array element is either 0, 1, or 2, where 0indicates a pixel/location that is predicted as background (i.e.,outside the object); 1 indicates a pixel/location that is a predictedobject border; and 2 indicates a pixel/location that is predicted objectinterior. For the cell segmentation model output, the object refers to acell. For the lymphocyte segmentation model, the object refers to alymphocyte. These masks array outputs may be temporarily buffered andprovided to processes 712 and 714, respectively.

Processes 712 and 714 receive the output mask array for the cellsegmentation (UNet) model and the output mask array for the lymphocytesegmentation (UNet) model, respectively. The processes 712 and 714 areperformed for each received tile and are used to express the informationin the mask arrays in terms of coordinates that are in the coordinatespace of the original whole-slide image.

In an example, the process 712 may access a stored image processinglibrary and use that library to find contours around the cell interiorclass, i.e., which corresponds to locations that have a 2 value in eachmask. In this way, the process 712 may perform a cell registrationprocess. The cell border class (denoted by locations with a 1 value ineach mask) ensures separation between neighboring cell interiors. Thisgenerates a list of every contour on each mask. Then, by treating eachcontour as a filled polygon, the process 712 determines the coordinatesof the contour's centroid (center of mass), from which the process 712produces a centroid list. Next, to generate outputs that are in thecoordinate space defined by the entire received image instead of thecoordinate space that is specific to a single tile in the image, eachcoordinate in the contour lists and the centroid lists is shifted.Without this shift, each coordinate would be in the coordinate space ofthe image tile that contains it. The value of each shift is equal to thecoordinates of the parent tile's upper-left corner on the receivedimage. In this example, the process 714 performs the same processes asthe process 712, but on the lymphocyte classes.

The processes 712 and 714 generate contour list outputs and centroidlist outputs, each corresponding to their respective UNet segmentationmodel. A contour is a set of coordinates that, when connected, outline adetected object. Each contour can be represented as a line of textcomposed of its constituent coordinates sequentially printed as pairs ofnumbers. Each contour list from the processes 712 and 714 may be savedas a text file composed of many such lines. The centroid lists are alist of pairs of numbers. Each of these outputs may be bufferedtemporarily and provided to process 716.

The process 716 receives tissue classification output (list of lists)from process 706, cell centroids and contour lists from process 712, andthe lymphocyte centroids and contour lists from process 714, andperforms cell segmentation integration.

For example, the process 716 may integrate the paired outputs of theprocesses 712 and 714 and produce a single concise list that containsthe most important information about cells. In an example, there are twomain components of the process 716.

In a first component of the process 716, information found in the cellsegmentation model and the lymphocyte segmentation model is combined.Before the information is combined, it exists as a list of cell contoursand a list of lymphocyte contours, but the lymphocyte contours are notnecessarily a subset of the cell contours because they are the outputsof two independent models (712 and 714). Therefore, it is desirable tomake the lymphocytes a subset of the cells because (1) biologically, ifan object is not a cell, it cannot be a lymphocyte, because lymphocytesare a type of cell (2) it is desirable to report percentages for datasets that have the same denominator. Therefore, the cell segmentationintegration process 716 may be performed by comparing the location ofeach cell with the location of every lymphocyte. (In one example, thismay be done only for the objects within a single tile, so the number ofcomparisons is not too high). In an example, if and only if a cell is“sufficiently close” to a lymphocyte, that cell is considered alymphocyte. The definition of “sufficiently close” may be established byempirically determining the median radius of objects detected by thelymphocyte segmentation model across a set of training histopathologyimages. Note this updated training image set (e.g., 403 in FIG. 4) isdifferent than the training set of images used to train the model asthis set of training histopathology images are annotated imagesgenerated by the model itself, resulting in orders of magnitudes greaternumbers of images, e.g., millions of automatically annotated images,that form a new or updated training set. Indeed, the training set forthe model may continue to grow with new received medical images thatsatisfy an accept/reject condition. This may be the case for tissueclassification model, as well as the cell segmentation and lymphocytesegmentation models. By generating a new training set from the model, asit assesses subsequent images, the model is able to (1) use the medianof millions of cells instead of just the ones outlined on the trainingtiles and (2) compare the actual size of objects that get detected, nothuman-drawn annotation sizes). Lymphocyte nuclei are typicallyspherical, so in an example all of these objects were modeled as circles(since they are two-dimensional slices of spheres). The radius of thesecircles was calculated, and the median value was used to determine thetypical size of lymphocyte detection. The result is that the final celllist is exactly the objects found by the cell segmentation model, whilethe purpose of the lymphocyte segmentation model is to provide a BooleanTrue/False label for each cell in that list.

In a second component of the process 716, each cell is binned into oneof the tissue classification tiles (from process 706) based on location.It is noted that in the example discussed here, the cell segmentationtiles may be of a different size than the tissue classification tilesbecause the models have different architectures. Nevertheless, theprocess 716 has the coordinates for each cell centroid, the coordinatesfor the top-left corner of each tissue classification tile, and the sizeof each tissue classification tile, and is configured to determine theparent tile for each cell based on its centroid location.

The process 716 generates an output that is a list of lists. Each nestedinterior list serves as nested classification that describes a singlecell, and contains the cell's centroid's coordinates, the tile number ofits parent tile, the tissue class of its parent tile, and whether thecell is classified as a lymphocyte or not. This information is listedfor each cell, and the output list is saved into the deep learningframework pipeline output json file.

Process 718, which may be implemented by a post-processing controllersuch as the biomarker metrics processing module 326, determines any of anumber of different biomarker metrics, in particular for this examplepredicted TILs status and other TILs metrics, as discussed.

For example, the process 718 may be configured to perform tissue areacalculations to determine the area covered by tissue, based on thetissue mask used in process 704. Because, in some examples, the tissuemask is a Boolean array that takes on values of 1 where tissue ispresent and 0 elsewhere, the process 718 may count the number of 1'sgives a measure of tissue area. This value is the number of squarepixels at 128× downsampling. Multiplying this by 16384 (i.e., for a128*128 tissue mask) gives the number of square pixels at nativeresolution (referred to as “x”). An image native resolution indicateshow many pixels are present per micron, and taking the square of thisnumber gives the number of square pixels per square micron (referred toas “y”). Dividing the number of square pixels at native resolution bythis resolution scaling factor (or, using the variables defined above,x/y) yields the number of square microns covered by tissue, and thus atissue area calculation. That is, the process 718 can generate afloating point number in [0, ∞) that indicates the tissue area in squaremicrons. This value may then be used in an accept/reject model processdiscussed below.

As an example of other biomarker statistics, the process 718 may befurther configured to perform a total nuclei calculation using the cellsegmentation integration output from process 716. For example, the totalnumber of nuclei on the slide is determined as the number of entries inthe cell segmentation integration output. The process 718 may alsoperform a tumor nuclei % calculation based on this output from theprocess 716. The total number of tumor nuclei on an image is the numberof entries in the cell segmentation integration output that satisfyrequirements: (i) the tissue class of the parent tile is tumor and (ii)the cell is not classified as a lymphocyte.

In addition to determining biomarker statistics, the process 718 may befurther configured to perform an accept/reject process based on thedetermined tissue area, total nuclei count, and tumor nuclei count. Inan example, the process 718 may be configured with a logistic regressionmodel, and these three variables are used as inputs, where the modeloutput is a binary recommendation for whether the slide should beaccepted for molecular sequencing or rejected. The logistic regressionmodel may be trained on a training set of images using these derivedvariables. For example, the training images may be formed of acceptedhistopathology images that were previously sent for sequencing, as wellas histopathology images that were rejected during routine pathologyreview. Alternatively, there may be a set threshold, for example, 20% ofnuclei on slide are tumor or a minimum number of tumor cells may berequired. In some examples, the model may also consider DNA ploidy ofthe tumor cells (data from karyotyping or DNA sequencing information)and may calculate an adjusted estimate of available genetic material bymultiplying the number of tumor nuclei by the average number of copiesof chromosomes detected in each tumor nucleus, divided by the normallyexpected copy number of 2. In some examples, the logistic regressionmodel may be configured to have three possible outputs (instead of two)by adding an uncertainty zone between accept and reject that recommendsmanual review. For example, the penultimate output of the logisticregression model is a real number, and the last step of the model would,in some examples, threshold this number at 0 to produce the binaryclassification. Instead, in some examples, an uncertainty zone isdefined as a range of numbers that includes 0; where values higher thanthis range correspond to rejection, values in the range correspond tomanual review, and values below this range correspond to acceptance. Theprocess 718 may be configured to calculate a size of this uncertaintyzone by performing a cross-validation experiment. For example, theprocess 718 may perform a training process repeated many times, butwhere in each repetition, a different random subset of the images in thetraining set are used. That will produce many final models that aresimilar but not identical, and the process 718 may use this variation todetermine the uncertainty range in the final logistic regression model.Therefore, the process 718 may generate a binary accept/reject output insome examples and an accept/reject/manual review output in someexamples.

Using the recommendation from the process 718, a decision is made. Forexample, the deep learning output post-processing controller maygenerate a report for images that are indicated as “accept” andautomatically send those images to a genomic sequencing system (112) formolecular sequencing, whereas images that are recommended “reject” arerejected and are not sent for molecular sequencing. If the “manualreview” option is configured and recommended, the image may be sent to apathologist or team of pathologists (118) to review the slide and todecide whether it should be sent for molecular sequencing or rejected.

FIG. 8 illustrates an example process 800 that may be executed by theimaging-based biomarker prediction system 102, the deep learningframework 300, or the deep learning framework 402, in particular in adeep learning framework having a single-scale configuration.

At a process 802, molecular training data is received at theimaging-based biomarker prediction system. This molecular training datais for a plurality of patients and may be obtained from a geneexpression dataset, such as from sources described herein. In someexamples, the molecular training data includes RNA sequence data. At ablock 804, the molecular training data is labeled by biomarker. One formof biomarker clustering includes labeling that may be performed bytaking pre-existing labels associated with the specimen, such as tumorsub-type, and associating the label with the molecular training data.Alternately, or in addition, the labeling may be performed byclustering, such as the use of an an automatic clustering algorithm. Oneexemplary algorithm, in the case of a CMS sub-type biomarker, is analgorithm to identify CMS sub-types in the molecular training data andcluster training data according to CMS sub-type. This automaticclustering may be performed within a deep learning frameworksingle-class classifier module, for example, or within a slide-levellabel pipeline, such as those in the deep learning framework 300. Insome examples, the molecular training data received at block 802 is RNAsequence data generated using an RNA wet lab, for example, and processedusing a bioinformatics pipeline.

In various embodiments, for example, each transcriptome data set may begenerated by processing a patient or tumor organoid sample through RNAwhole exome next generation sequencing (NGS) to generate RNA sequencingdata, and the RNA sequencing data may be processed by a bioinformaticspipeline to generate a RNA-seq expression profile for each sample. Thepatient sample may be a tissue sample or blood sample containing cancercells. RNA may be isolated from blood samples or tissue sections usingcommercially available reagents, for example, proteinase K, TURBODNase-I, and/or RNA clean XP beads. The isolated RNA may be subjected toa quality control protocol to determine the concentration and/orquantity of the RNA molecules, including the use of a fluorescent dyeand a fluorescence microplate reader, standard spectrofluorometer, orfilter fluorometer.

cDNA libraries may be prepared from the isolated RNA, purified, andselected for cDNA molecule size selection using commercially availablereagents, for example Roche KAPA Hyper Beads. cDNA library preparationmay include reverse transcription. In another example, a New EnglandBiolabs (NEB) kit may be used. cDNA library preparation may include theligation of adapters onto the cDNA molecules. For example, UDI adapters,including Roche SeqCap dual end adapters, or UMI adapters (for example,full length or stubby Y adapters) may be ligated to the cDNA molecules.In this example, adapters are nucleic acid molecules that may serve asbarcodes to identify cDNA molecules according to the sample from whichthey were derived and/or to facilitate the downstream bioinformaticsprocessing and/or the next generation sequencing reaction. The sequenceof nucleotides in the adapters may be specific to a sample in order todistinguish between sequencing data obtained for different samples. Theadapters may facilitate the binding of the cDNA molecules to anchoroligonucleotide molecules on the sequencer flow cell and may serve as aseed for the sequencing process by providing a starting point for thesequencing reaction.

cDNA libraries may be amplified and purified using reagents, forexample, Axygen MAG PCR clean up beads. Amplification may includepolymerase chain reaction (PCR) techniques, which are distinct fromquantitative or reverse transcription quantitative PCR (qPCR orRT-qPCR). Then the concentration and/or quantity of the cDNA moleculesmay be quantified using a fluorescent dye and a fluorescence microplatereader, standard spectrofluorometer, or filter fluorometer.

cDNA libraries may be pooled and treated with reagents to reduceoff-target capture, for example Human COT-1 and/or IDT xGen UniversalBlockers, before being dried in a vacufuge. Pools may then beresuspended in a hybridization mix, for example, IDT xGen Lockdown, andprobes may be added to each pool, for example, IDT xGen Exome ResearchPanel v1.0 probes, IDT xGen Exome Research Panel v2.0 probes, other IDTprobe panels, Roche probe panels, or other probes. Pools may beincubated in an incubator, PCR machine, water bath, or other temperaturemodulating device to allow probes to hybridize. Pools may then be mixedwith Streptavidin-coated beads or another means for capturing hybridizedcDNA-probe molecules, especially cDNA molecules representing exons ofthe human genome. In another embodiment, polyA capture may be used.Pools may be amplified and purified once more using commerciallyavailable reagents, for example, the KAPA HiFi Library Amplification kitand Axygen MAG PCR clean up beads, respectively.

The cDNA library may be analyzed to determine the concentration orquantity of cDNA molecules, for example by using a fluorescent dye (forexample, PicoGreen pool quantification) and a fluorescence microplatereader, standard spectrofluorometer, or filter fluorometer. The cDNAlibrary may also be analyzed to determine the fragment size of cDNAmolecules, which may be done through gel electrophoresis techniques andmay include the use of a device such as a LabChip GX Touch. Pools may becluster amplified using a kit (for example, Illumina Paired-end ClusterKits with PhiX-spike in). In one example, the cDNA library preparationand/or whole exome capture steps may be performed with an automatedsystem, using a liquid handling robot (for example, a SciClone NGSx).

The library amplification may be performed on a device, for example, anIllumina C-Bot2, and the resulting flow cell containing amplifiedtarget-captured cDNA libraries may be sequenced on a next generationsequencer, for example, an Illumina HiSeq 4000 or an Illumina NovaSeq6000 to a unique on-target depth selected by the user, for example,300×, 400×, 500×, 10,000×, etc. The next generation sequencer maygenerate a FASTQ, BCL, or other file for each patient sample or eachflow cell.

If two or more patient samples are processed simultaneously on the samesequencer flow cell, reads from multiple patient samples may becontained in the same BCL file initially and then divided into aseparate FASTQ file for each patient. A difference in the sequence ofthe adapters used for each patient sample could serve the purpose of abarcode to facilitate associating each read with the correct patientsample and placing it in the correct FASTQ file.

Each FASTQ file contains reads that may be paired-end or single reads,and may be short-reads or long-reads, where each read shows one detectedsequence of nucleotides in an mRNA molecule that was isolated from thepatient sample, inferred by using the sequencer to detect the sequenceof nucleotides contained in a cDNA molecule generated from the isolatedmRNA molecules during library preparation. Each read in the FASTQ fileis also associated with a quality rating. The quality rating may reflectthe likelihood that an error occurred during the sequencing procedurethat affected the associated read.

Each FASTQ file may be processed by a bioinformatics pipeline. Invarious embodiments, the bioinformatics pipeline may filter FASTQ data.Filtering FASTQ data may include correcting sequencer errors andremoving (trimming) low quality sequences or bases, adapter sequences,contaminations, chimeric reads, overrepresented sequences, biases causedby library preparation, amplification, or capture, and other errors.Entire reads, individual nucleotides, or multiple nucleotides that arelikely to have errors may be discarded based on the quality ratingassociated with the read in the FASTQ file, the known error rate of thesequencer, and/or a comparison between each nucleotide in the read andone or more nucleotides in other reads that has been aligned to the samelocation in the reference genome. Filtering may be done in part or inits entirety by various software tools. FASTQ files may be analyzed forrapid assessment of quality control and reads, for example, by asequencing data QC software such as AfterQC, Kraken, RNA-SeQC, FastQC,(see Illumina, BaseSpace Labs orhttps://www.illumina.com/products/by-type/informatics-products/basespace-sequence-hub/apps/fastqc.html),or another similar software program. For paired-end reads, reads may bemerged.

For each FASTQ file, each read in the file may be aligned to thelocation in the reference genome having a sequence that best matches thesequence of nucleotides in the read. There are many software programsdesigned to align reads, for example, Bowtie, Burrows Wheeler Aligner(BWA), programs that use a Smith-Waterman algorithm, etc. Alignment maybe directed using a reference genome (for example, GRCh38, hg38, GRCh37,other reference genomes developed by the Genome Reference Consortium,etc.) by comparing the nucleotide sequences in each read with portionsof the nucleotide sequence in the reference genome to determine theportion of the reference genome sequence that is most likely tocorrespond to the sequence in the read. The alignment may take RNAsplice sites into account. The alignment may generate a SAM file, whichstores the locations of the start and end of each read in the referencegenome and the coverage (number of reads) for each nucleotide in thereference genome. The SAM files may be converted to BAM files, BAM filesmay be sorted, and duplicate reads may be marked for deletion.

In one example, kallisto software may be used for alignment and RNA readquantification (see Nicolas L Bray, Harold Pimentel, Pall Melsted andLior Pachter, Near-optimal probabilistic RNA-seq quantification, NatureBiotechnology 34, 525-527 (2016), doi:10.1038/nbt.3519). In analternative embodiment, RNA read quantification may be conducted usinganother software, for example, Sailfish or Salmon (see Rob Patro,Stephen M. Mount, and Carl Kingsford (2014) Sailfish enablesalignment-free isoform quantification from RNA-seq reads usinglightweight algorithms. Nature Biotechnology (doi:10.1038/nbt.2862) orPatro, R., Duggal, G., Love, M. I., Irizarry, R. A., & Kingsford, C.(2017). Salmon provides fast and bias-aware quantification of transcriptexpression. Nature Methods.). These RNA-seq quantification methods maynot require alignment. There are many software packages that may be usedfor normalization, quantitative analysis, and differential expressionanalysis of RNA-seq data.

For each gene, the raw RNA read count for a given gene may becalculated. The raw read counts may be saved in a tabular file for eachsample, where columns represent genes and each entry represents the rawRNA read count for that gene. In one example, kallisto alignmentsoftware calculates raw RNA read counts as a sum of the probability, foreach read, that the read aligns to the gene. Raw counts are thereforenot integers in this example.

Raw RNA read counts may then be normalized to correct for GC content andgene length, for example, using full quantile normalization and adjustedfor sequencing depth, for example, using the size factor method. In oneexample, RNA read count normalization is conducted according to themethods disclosed in U.S. patent application Ser. No. 16/581,706 orPCT19/52801, titled Methods of Normalizing and Correcting RNA ExpressionData and filed Sep. 24, 2019, which are incorporated by reference hereinin their entirety. The rationale for normalization is the number ofcopies of each cDNA molecule in the sequencer may not reflect thedistribution of mRNA molecules in the patient sample. For example,during library preparation, amplification, and capture steps, certainportions of mRNA molecules may be over or under-represented due toartifacts that arise during various aspects of priming of reversetranscription caused by random hexamers, amplification (PCR enrichment),rRNA depletion, and probe binding and errors produced during sequencingthat may be due to the GC content, read length, gene length, and othercharacteristics of sequences in each nucleic acid molecule. Each raw RNAread count for each gene may be adjusted to eliminate or reduce over- orunder-representation caused by any biases or artifacts of NGS sequencingprotocols. Normalized RNA read counts may be saved in a tabular file foreach sample, where columns represent genes and each entry represents thenormalized RNA read count for that gene.

A transcriptome value set may refer to either normalized RNA read countsor raw RNA read counts, as described above.

Returning to FIG. 8, at a block 804, the molecular training data (e.g.,such RNA sequence data) is labeled by biomarker and clustered, using anautomatic clustering algorithm, such as an algorithm to identify CMSsub-types in the molecular training data and cluster training dataaccording to CMS sub-type. This automatic clustering may be performedwithin a deep learning framework single-class classifier module, forexample, or within a slide-level label pipeline, such as those in thedeep learning framework 300.

At a block 806, for each biomarker cluster (each corresponding to adifferent biomarker, such as a different CMS sub-type or HRD),histopathology images from the associated patients are obtained. Thesehistopathology images may be H&E slide images having a slide-levellabel, for example. At a block 808, for each biomarker cluster, theselabeled histopathology images are provided to the deep learningframework for training biomarker classification models, such as multipleCMS classification models to predict different CMS sub-types. A set oftrained biomarker classifiers (classification models) are generated atblock 810 as a result. In this way, the blocks 802-810 representing atraining process.

A prediction process starts at a block 812, where a new (unlabeled orlabeled) histopathology image, such as a H&E slide image, is receivedand provided to the single-scale biomarker classifiers generated byblock 810, and a block 814 predicts biomarker classification on thereceived histopathology image as determined by the one or more biomarkerclassification models, such as one or more CMS sub-types or HRD.

As with block 610, new histopathology images may be received at theblock 814 from the physical clinical records system or primary caresystem and applied to a trained deep learning framework which appliesits tissue classification model and/or biomarker classification modelsdetermines a biomarker prediction. That prediction score can bedetermined for an entire histopathology image, for example.

Further, as with the process 600, as shown in process 900 of FIG. 9,after prediction, the predicted biomarker classification from block 814may be received at block 902. A clinical report for the histopathologyimage, and thus for the patient, may be generated at the block 904including predicted biomarker status and, at the block 906, an overlaymap may be generated showing the predicted biomarker status for displayto a clinician or for providing to a pathologist for determining apreferred immunotherapy corresponding to the predicted biomarker.

FIGS. 10A and 10B illustrate examples of a digital overlay maps createdby the overlay map generator 324 of system 300, for example. Theseoverlay maps may be generated as static digital reports displayed toclinicians or as dynamic reports allowing user interaction through agraphical user interface (GUI). FIG. 10A illustrates a tissue classoverlay map generated by the overlay map generator 324. FIG. 10Billustrates a cell outer edge overlay map generated by the overlay mapgenerator 324.

In an example, the overlay map generator 324 may display the digitaloverlays as transparent or opaque layers that cover the histopathologyimage, aligned such that the image location shown in the overlay and thehistopathology image are in the same location on the display. Theoverlay map may have varying degrees of transparency. The degree oftransparency may be adjustable by the user, in a dynamic reporting modeof the overlay map generator 324. The overlay map generator 326 mayreport the percentage of the labeled tiles that are associated with eachtissue class label, ratios of the number of tiles classified under eachtissue class, the total area of all grid tiles classified as a singletissue class, and ratios of the areas of tiles classified under eachtissue class. The overlay map may be displayed as a heatmap showingdifferent tissue classifications and having different pixel intensitylevels that correspond to different biomarker status levels, e.g., inthe TILs example, showing higher intensity pixels for tissue regionshaving higher predicted TILs status (higher %) and lower intensitypixels for tissue regions having lower predicted TILs status (lower %).

In an example, the deep learning output post-processing controller 308may also report the total number of cells or the percentage of cellsthat are located in an area defined by either a user, the entire slide,a single grid tile, by all grid tiles classified under each tissueclass, or cells that are classified as immune cells. The controller 308may also report the number of cells classified as lymphocyte cells thatare located within areas classified as tumor or any other tissue class.

In an example, the digital overlays and reports generated by thecontroller 308 may be used to assist medical professionals in moreaccurately estimating tumor purity, and in locating regions or diagnosesof interest, including invasive tumors having tumor cells that protrudeinto the non-tumor tissue region that surrounds the tumor. They can alsoassist medical professionals in prescribing treatments. For example, thenumber of lymphocytes in areas classified as tumor may predict whetherimmunotherapy will be successful in treating a patient's cancer.

In an example, the digital overlays and reports generated by thecontroller 308 may also be used to determine whether the slide samplehas enough high-quality tissue for successful genetic sequence analysisof the tissue, for example implementing an accept/reject/manualdetermination as discussed in process 700. Genetic sequence analysis ofthe tissue on a slide is likely to be successful if the slide containsan amount of tissue and/or has a tumor purity value that exceeds auser-defined tissue amount and tumor purity thresholds. The controller308 may use the process 700 to label a slide as accepted or rejected forsequence analysis, depending on the amount of tissue present on theslide and the tumor purity of the tissue on the slide. The controller308 may also label a slide as uncertain, according to the process 700,as well, using a user-defined tissue amount threshold and a user-defineduncertainty range obtained from a user interacting with the digitaloverlays and reports from the generator 324.

In an example, the controller 308, implementing the process 700, e.g.,using the biomarker metrics processing module 326, calculates the amountof tissue on a slide by measuring the total area covered by the tissuein the histopathology image or by counting the number of cells on theslide. The number of cells on the slide may be determined by the numberof cell nuclei visible on the slide. In an example, the controller 308calculates the proportion of tissue that is cancer cells by dividing thenumber of cell nuclei within grid areas that are labeled tumor by thetotal number of cell nuclei on the slide. The controller 308 may excludecell nuclei or outer edges of cells that are located in tumor areas butwhich belong to cells that are characterized as lymphocytes. Theproportion of tissue that is cancer cells is known as the tumor purityof the sample. The controller 308 then compares the tumor purity to theuser-selected minimum tumor purity threshold and the number of cells inthe digital image to a user-selected minimum cell threshold (as input bya user interacting with the overlay map generator 324) and approves thetissue slide depicted in the image for molecular testing, includinggenetic sequence analysis, if both thresholds are exceeded. In oneexample, the user-selected minimum tumor purity threshold is 0.20, whichis 20%. Although any number of tumor purity thresholds may be selected,including 1%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 45%, 50%, orhigher.

In another example, the controller 308 gives the image a compositetissue amount score that multiplies the total area covered by tissuedetected on the slide by a first multiplier value, multiplies the numberof cells counted on the slide by a second multiplier value, and sums theproducts of these multiplications.

In an example, the controller 308 may calculate whether the grid areasthat are labeled tumor are spatially consolidated or dispersed amongnon-tumor grid areas. If the controller 308 determines that the tumorareas are spatially consolidated, the overlay map generator 324 mayproduce a digital overlay of a recommended cutting boundary thatseparates the image regions classified as tumor and the image regionsclassified as non-tumor or within the areas classified as non-tumor,proximal to the areas classified as tumor. This recommended cuttingboundary can be a guide to assist a technician in dissecting a slide toisolate a maximum amount of tumor or non-tumor tissue from the slide,especially for molecular testing, including genetic sequence analysis.

In an example, the controller 308 may include clustering algorithms thatcalculate and report information about the spacing and density of typeclassified cells, tissue class classified tiles, or visually detectablefeatures on the slide. The spacing information includes distributionpatterns and heat maps for lymphocytes, immune cells, tumor cells, orother cells. These patterns may include clustered, dispersed, dense, andnon-existent. This information is useful to determine whether immunecells and tumor cells cluster together and what percentage of thecluster areas overlay, which may facilitate in predicting immuneinfiltration and patient response to immunotherapy.

The controller 308 may also calculate and report average tumor cellroundness, average tumor cell perimeter length, and average tumor nucleidensity.

The spacing information also includes mixture levels of tumor cells andimmune cells. The clustering algorithms can calculate the probabilitythat two adjacent cells on a given slide or in a region of a slide willbe either two tumor cells, two immune cells, or one tumor cell and oneimmune cell.

The clustering algorithms may also measure the thickness of any stromapattern located around an area classified as tumor. The thickness ofthis stroma surrounding the tumor region may be a predictor of apatient's response to treatment.

In an example, the controller 308 may also calculate and reportstatistics including mean, standard deviation, sum, etc. for thefollowing information in each grid tile of either a single slide imageor aggregated from many slide images: red green blue (RGB) value,optical density, hue, saturation, grayscale, and stain deconvolution.Deconvolution includes the removal of the visual signal created by anyindividual stain or combination of stains, including hematoxylin, eosin,or IHC staining.

The controller 308 may also incorporate known mathematical formulae fromthe fields of physics and image analysis to calculate visuallydetectable basic features for each grid tile. Visually detectable basicfeatures, including lines, patterns of alternating brightness, andoutlineable shapes, may be combined to create visually detectablecomplex features including cell size, cell roundness, cell shape, andstaining patterns referred to as texture features.

In other examples, the digital overlays, reports, statistics, andestimates produced by the overlay map generator 324 may be useful forpredicting patient survival, patient response to a specific cancertreatment, PD-L1 status of a tumor or immune cluster, microsatelliteinstability (MSI), tumor mutational burden (TMB), and the origin of atumor when the origin is unknown or the tumor is metastatic. Thebiomarker metrics processing module 326 may also calculate quantitativemeasurements of predicted patient survival, patient response to aspecific cancer treatment, PD-L1 status of a tumor or immune cluster,microsatellite instability (MSI), and tumor mutational burden (TMB).

In an example, the controller 308 may calculate relative densities ofeach type of immune cell on an entire slide, in the areas designated astumor or another tissue class. Immune tissue classes includelymphocytes, cytotoxic T cells, B cells, NK cells, macrophages, etc.

In an example, the act of scanning or otherwise digitally capturing ahistopathology slide automatically triggers the deep learning framework300 to analyze the digital image of that histopathology slide.

In an example, the overlay map generator 324 allows a user to edit acell outer edge or a border between two tissue classes on a tissue classoverlay map or a cell outer edge overlay map and saves the altered mapas a new overlay.

FIG. 11 illustrates a process 1100 for preparing digital images ofhistopathology slides for tissue classification, biomarker detection,and mapping analysis, as may be implemented using the system 300. Theprocess 1100 may be performed on each received image for analysis andbiomarker prediction. In some examples, the process 1100 may beperformed, in whole or in part, on initially received training images.Each of the processes described in FIG. 9 may be performed by thepre-processing controller 302, where any one or more of the processesmay be performed by the normalization module 310 and/or the tissuedetector 314.

In an example, such as when training a classifier model, each digitalimage file received by the pre-processing controller 302, at 1102,contains multiple versions of the same image content, and each versionhas a different resolution. The file stores these copies in stackedlayers, arranged by resolution such that the highest resolution imagecontaining the greatest number of bytes is the bottom layer. This isknown as a pyramidal structure. In one example, the highest resolutionimage is the highest resolution achievable by the scanner or camera thatcreated the digital image file.

In an example, each digital image file also contains metadata thatindicates the resolution of each layer. The pre-processing controller302, at process 1104, can detect the resolution of each layer in thismetadata and compare it to user-selected resolution criteria to select alayer with optimal resolution for analysis. In one example, the optimalresolution is 1 pixel per micron (downsampled by 4).

In an example, the pre-processing controller 302 receives a Tagged ImageFile Format (TIFF) file with a bottom layer resolution of four pixelsper micron. This resolution of 4 pixels per micron corresponds to theresolution achieved by a microscope objective lens with a magnificationpower of “40×”. In one example, the area that may have tissue on theslide is up to 100,000×100,000 pixels in size.

In an example, the TIFF file has approximately 10 layers, and theresolution of each layer is half as high as the resolution of the layerbelow it. If the higher resolution layer had a resolution of four pixelsper micron, the layer above it will have two pixels per micron. The arearepresented by one pixel in the upper layer will be the size of the arearepresented by four pixels in the lower layer, meaning that the lengthof each side of the area represented by one upper layer pixel will betwice the length of each side of the area represented by one lower layerpixel.

Each layer may be a 2× downsampling of the layer below it, as performedat process 1106. Downsampling is a method by which a new version of anoriginal image can be created with a lower resolution value than theoriginal image. There are many methods known in the art fordownsampling, including nearest-neighbor, bilinear, hermite, bell,Mitchell, bicubic, and Lanczos resampling.

In an example, 2× downsampling means that the red green blue (RGB)values from three of four pixels that are located in a square in thehigher resolution layer are replaced by the RGB value from the fourthpixel to create a new, larger pixel in the layer above, which occupiesthe same space as the four averaged pixels.

In an example, the digital image file does not contain a layer or animage with the optimal resolution. In this case, the pre-processingcontroller 302 can receive an image from the file having a resolutionthat is higher than the optimal resolution and downsample the image at aratio that achieves the optimal resolution, at process 1106.

In an example, the optimal resolution is 2 pixels per micron, or “20×”magnification, but the bottom layer of a TIFF file is 4 pixels permicron and each layer is downsampled 4× compared to the layer below it.In this case, the TIFF file has one layer at 40× and the next layer at10× magnification, but does not have a layer at 20× magnification. Inthis example, the pre-processing controller 302 reads the metadata andcompares the resolution of each layer to the optimal resolution and doesnot find a layer with the optimal resolution. Instead, thepre-processing controller 302 retrieves the 40× magnification layer,then downsamples the image in that layer at a 2× downsampling ratio tocreate an image with the optimal resolution of 20× magnification.

Also at process 1106, after the pre-processing controller 302 obtains animage with an optimal resolution, it locates all parts of the image thatdepict tumor sample tissue and digitally eliminates debris, pen marks,and other non-tissue objects.

In an example, also at process 1106, the pre-processing controller 302differentiates between tissue and non-tissue regions of the image anduses Gaussian blur removal to edit pixels with non-tissue objects. In anexample, any control tissue on a slide that is not part of the tumorsample tissue can be detected and labeled as control tissue by thetissue detector or manually labeled by a human analyst as control tissuethat should be excluded from the downstream tile grid projections.

Non-tissue objects include artifacts, markings, and debris in the image.Debris includes keratin, severely compressed or smashed tissue thatcannot be visually analyzed, and any objects that were not collectedwith the sample.

In an example, also at process 1106, a slide image contains marker inkor other writing that the controller 302 detects and digitally deletes.Marker ink or other writing may be transparent over the tissue, meaningthat the tissue on the slide may be visible through the ink. Because theink of each marking is one color, the ink causes a consistent shift inthe RGB values of the pixels that contain stained tissue underneath theink compared to pixels that contain stained tissue without ink.

In an example, also at process 1106, the controller 302 locates portionsof the slide image that have ink by detecting portions that have RGBvalues that are different from the RGB values of the rest of the slideimage, where the difference between the RGB values from the two portionsis consistent. Then, the tissue detector may subtract the differencebetween the RGB values of the pixels in the ink portions and the pixelsin the non-ink portions from the RGB values of the pixels in the inkportions to digitally delete the ink.

In an example, also at process 1106, the controller 302 eliminatespixels in the image that have low local variability. These pixelsrepresent artifacts, markings, or blurred areas caused by the tissueslice being out of focus, an air bubble being trapped between the twoglass layers of the slide, or pen marks on the slide.

In an example, also at process 1106, the controller 302 removes thesepixels by converting the image to a grayscale image, passing thegrayscale image through a Gaussian blur filter that mathematicallyadjusts the original grayscale value of each pixel to a blurredgrayscale value to create a blurred image. Other filters may be used toblur the image. Then, for each pixel, the controller 302 subtracts theblurred grayscale value from the original grayscale value to create adifference grayscale value. In one example, if a difference grayscalevalue of a pixel is less than a user-defined threshold, it may indicatethat the blur filter did not significantly alter the original grayscalevalue and the pixel in the original image was located in a blurredregion. The difference grayscale values may be compared to a thresholdto create a binary mask that indicates where the blurred regions arethat may be designated as non-tissue regions. A mask may be a copy of animage, where the colors, RGB values, or other values in the pixels areadjusted to show the presence or absence of an object of a certain typeto show the location of all objects of that type. For example, thebinary mask may be generated by setting the binary value of each pixelto 0 if the pixel has a difference grayscale value less than auser-defined blur threshold and setting the binary value of each pixelto 1 if the pixel has a difference grayscale value higher than or equalto a user-defined blur threshold. The regions of the binary mask thathave pixel binary values of 0 indicate blurred areas in the originalimage that may be designated as non-tissue.

The controller 302 may also mute or remove extreme brightness ordarkness in the image, at process 1108. In one example, the controller302 converts the input image to a grayscale image, and each pixelreceives a numerical value according to how bright the pixel is. In oneexample, the grayscale values range from 0 to 255, where 0 representsblack and 255 represents white. In pixels with a grayscale value above abrightness threshold value, the tissue detector will replace thegrayscale value of those pixels with the brightness threshold value. Forpixels with a grayscale value below a darkness threshold value, thetissue detector will replace the grayscale value of those pixels equalwith the darkness threshold value. In one example, the brightnessthreshold value is approximately 210. In one example, the darknessthreshold value is approximately 45. The tissue detector stores theimage with the new grayscale values in a data file.

In an example, the controller 302 analyzes the altered image for anyartifacts, debris, or markings that remain after the first analysis, atprocess 1110. The tissue detector scans the image and categorizes anyremaining groups of pixels with a certain color, size, or smoothness asnon-tissue.

In an example, the slide has H&E staining and most tissue in thehistopathology image will have a pink stain. In this example, thecontroller 302 categorizes all objects without any pink or red hue, asdetermined by the RGB value of the pixels that represent the object, asnon-tissue. The tissue detector 314 may interpret any color or the lackof any color in a pixel to indicate the presence or absence of tissue inthat pixel.

In an example, the controller 302 detects the contours of each object inthe image in order to measure the size and smoothness of each object.Pixels that are very dark may be debris, and pixels that are very brightmay be background, which are both non-tissue objects. Therefore, thecontroller 302 may detect the contours of each object by converting theimage to grayscale, comparing the grayscale values of each pixel to arange of user-determined range of values that are not too bright or toodark, and determining whether the grayscale value is within the range toproduce a binary image where each pixel is assigned one of two numericalvalues.

For example, to threshold an image, the controller 302 may compare thegrayscale values of each pixel to a user-defined range of values andreplace each grayscale value outside of the user-defined range with thevalue 0 and each grayscale value within a user-defined range with thevalue 1. Then, the controller 302 draws all contours of all objects asthe outer edge of each group of adjacent pixels having a value of 1.Closed contours indicate the presence of an object, and the controller302 measures the area within the contours of each object to measure thesize of the object.

In an example, tissue objects on a slide are unlikely to make contactwith the outer edges of the slide so the controller 302 categorizes allobjects that contact the edge of a slide as non-tissue.

In an example, after measuring the size of each object, the controller302 ranks the sizes of all objects and designates the largest value tobe the size of the largest object. The controller 302 divides the sizeof each object by the size of the largest object and compares theresulting size quotient to a user-defined size threshold value. If thesize quotient for an object is smaller than the user-defined sizethreshold value, the controller 302 designates that object asnon-tissue. In one example, the user-defined size threshold value is0.1.

Before measuring the size of each object, at process 1106, thecontroller 302 may first downsample the input image to reduce thelikelihood of designating portions of a tissue object as non-tissue. Forexample, a single tissue object may appear as a first tissue objectportion surrounded by one or more additional tissue object portionshaving a smaller size. After thresholding, the additional tissue objectportions may have a size quotient smaller than the user-defined sizethreshold value and may be erroneously designated as non-tissue.Downsampling before thresholding causes a small group of adjacent pixelshaving values of 1 surrounded by pixels having values of 0 in theoriginal image to be included in a proximal, larger group of pixelshaving values of 1. The opposite may also be true, for small groups ofadjacent pixels having values of 0 surrounded by pixels having values of1 in the original image to be included in a proximal, larger group ofpixels having values of 0.

In an example, the controller 302 downsamples an image having 40×magnification by a ratio of 16×, so the magnification of the resultingdownsampled image is 40/16× and each pixel in the downsampled imagerepresents 16 pixels in the original image.

In an example, at process 1110, the controller 302 detects theboundaries of each object on the slide as a cluster of pixels havingbinary or RGB values that do not equal zero, surrounded by pixels withRGB values equal to zero, indicating an object border. If the pixelsforming the boundaries lie on a relatively straight line, the controller302 classifies the object as non-tissue. For example, the controller 302outlines a shape with a closed polygon. If the number of vertices of thepolygon is less than a user-defined minimum vertices threshold, thepolygon is deemed to be a simple, inorganic shape that is too smooth,and marked as non-tissue. The controller 302 then applies a tilingprocess to the normalized image, at process 1112.

FIGS. 12A-12C illustrate an example architecture 1200 that may be usedfor the classification models of the module 306. For example, The samearchitecture 1200 may be used for each of the tissue segmentation model322 and tissue classification model 320, both implemented using an FCNconfiguration or any neural network herein. The tissue classifier module306 includes a tissue classification algorithm (see FIGS. 12A-12C) thatassigns a tissue class label to the image represented in each receivedtile (example tiles 1302 are labeled in a first portion 1304 of ahistopathology image 1300, shown in FIG. 13). In an example, the overlaymap generator 324 may report the assigned tissue class label associatedwith each small square tile by displaying a grid-based digital overlaymap in which each tissue class is represented by a unique color (seeFIG. 12A).

A smaller tile size may cause an increase in the amount of time requiredfor the tissue classifier module 306 to analyze the input image.Alternatively, a larger tile size may increase the likelihood that atile will contain more than one tissue class and make it difficult toassign a single tissue class label to the tile. In this case, thearchitecture 1200 may calculate an equal probability for two or moretissue class labels being accurately assigned to a single small squaretile instead of calculating that one of the tissue class labels has ahigher probability of describing the image in the small square tile,compared to the other tissue class labels.

In an example, each side of each small square tile is approximately 32microns long and approximately 5-10 cells fit in each small square tile.This small tile size allows the tissue classifier module 306 to createmore spatially accurate borders when determining the boundary betweentwo neighboring small square tile regions that depict two distincttissue classes. In one example, each side of the small square tile canbe as short as 1 micron.

In an example, the size of each tile may be set by the user to contain aspecific number of pixels. In this example, the resolution of the inputimage will determine the length of each side of the tile, as measured inmicrons. At different resolutions, the micron length of the tile sidewill vary and the number of cells in each tile may vary.

The architecture 1200 recognizes various pixel data patterns in theportion of the digital image that is located within or near each smallsquare tile and assigns a tissue class label to each small square tilebased on those detected pixel data patterns. In one example, a mediumsquare tile centered around a small square tile contains the area of aslide image that is close enough to the small square tile to contributeto the label assignment for that small square tile.

In an example, each side of a medium square tile is approximately 466microns long, and each medium square tile contains approximately 225(15×15) small square tiles. In one example, this medium tile sizeincreases the likelihood that structural tissue features can fit withina single medium tile and provide context to the algorithm when labelingthe central small square tile. Structural tissue features may includeglands, ducts, vessels, immune clusters, etc.

In an example, this medium tile size is selected such that it can negatethe shrinkage that occurs during convolution.

During convolution with the architecture 1200, an input image matrix ismultiplied by a filter matrix to create a result matrix, and shrinkagerefers to a case where the result matrix is smaller than the input imagematrix. The dimensions of a filter matrix in a convolution layer affectsthe number of rows and columns lost to shrinkage. The total number ofmatrix entries that are lost to shrinkage by processing an image througha particular CNN can be calculated depending on the number ofconvolution layers in the CNN and the dimensions of the filter matricesin each convolution layer. (See FIGS. 12A-12C)

In the example shown in FIG. 12B, the convolution layers in combinationlose 217 total matrix rows or columns from the top, bottom, and two sideedges of the matrix, so the medium square tile is set to equal the smallsquare tile plus 217 pixels on each side of the small square tile.

In an example, two neighboring small square tiles share a side and areeach at the center of a medium square tile. The two medium square tilesoverlap. Of the 466*466 small pixels located in each medium square tile,the two medium square tiles will share all but 32*466 pixels. In oneexample, each convolution layer of the algorithm (see FIGS. 12A and 12B)analyzes both medium square areas simultaneously such that the algorithmproduces two vectors of values (one for each of the two small squaretiles).

The vector of values contains a probability value for each tissue classlabel, indicating the likelihood that the small square tile depicts thattissue class. The vectors of values are/may be arranged in a matrix, toform a 3-dimensional probability data array. The location of each vectorin the 3-dimensional probability data array, relative to the othervectors, will correspond to the location of the associated small squaretile, relative to the other small square tiles included in the algorithmanalysis.

In the example, 434×434 (188,356) of the 466×466 (217,156) pixels ineach medium square tile are common to both medium square tiles. Byanalyzing both medium square tiles simultaneously, the algorithmincreases efficiency.

In one example, the architecture 1200 can further increase efficiency byanalyzing a large tile formed by multiple overlapping medium squaretiles, each of which contains many small square tiles surrounding onecenter small square tile that receives a tissue class label. In thisexample, the algorithm still generates one data structure in the form ofa 3-dimensional probability data array containing one vector ofprobabilities for each small square tile, wherein the location of thevector within the 3-dimensional array corresponds to the location of thesmall tile within the large tile.

The architecture 1200, e.g., in the tissue classifier module 306, savesthis 3-dimensional probability data array, and the overlay map generator324 converts the tissue class label probabilities for each small squaretile into a tissue class overlay map. In an example, the overlay mapgenerator 324 may compare the probabilities stored in each vector todetermine the largest probability value associated with each smallsquare tile. The tissue class label associated with that largest valuemay be assigned to that small square tile and only the assigned labelswill be displayed in the tissue class overlay map.

In an example, matrices generated by each layer of the architecture 1200for the large square tile are stored in graphics processing unit (GPU)memory. The capacity of the GPU memory and the amount of GPU memoryrequired for each entry in the 3-dimensional probability data array maydetermine the maximum possible size of the large square tile. In oneexample, the GPU memory capacity is 250 MB and each entry in thematrices requires 4 bytes of GPU memory. This allows a large tile sizeof 4,530 pixels by 4,530 pixels, calculated as follows: 4bytes/entry*4530*4530*3 entries for each large tile=246 (˜250) MB of GPUmemory required per large square tile. In another example, each entry inthe matrices requires 8 bytes of GPU memory. In this example, a 16 GBGPU can process 32 large tiles simultaneously, each large tile havingdimensions of 4,530 pixels by 4,530 pixels, calculated as follows: 32large tiles*8 bytes/entry*4530*4530*3 entries for each large tile=14.7(˜16) GB of GPU memory required.

In an example, each entry in the 3-dimensional probability data array isa single precision floating-point format (float32) data entry.

In an example, there are 16,384 (1282) non-overlapping small squaretiles that form a large square tile. Each small square tile is thecenter of a medium square tile having sides that are each approximately466 pixels long. The small square tiles form a center region of a largesquare tile having sides that are each approximately 4,096 pixels long.The medium square tiles all overlap and create a border around all foursides of the center region that is approximately 217 pixels wide.Including the border, each large square tile has sides that are eachapproximately 4,530 pixels long.

In this example, this large square tile size allows simultaneouscalculations that reduce the redundant computation percentage by 99%.This may be calculated as follows: first, select a pixel on the interiorof a large square tile (any pixel at least 434 pixels from the edge ofthe large square tile); construct a region that is the size of a mediumsquare tile (466 pixels per edge) with this model pixel at the center;and then for any small square tile centered within this constructedregion, the model pixel is contained within that small square tile'scorresponding medium square tile. There are (466/32){circumflex over( )}2=˜217 such small square tiles within the large square tile. Forpixels not on the interior of the large square tile, the number of smallsquare tiles that satisfy this condition is smaller. The numberdecreases linearly as the distance between the selected small squaretile and the edge of the large square tile decreases, then again as thedistance between the selected small square tile and the cornerdecreases, where a small number of pixels (˜0.005%) only contributetowards the classification of a single small square tile. Performingclassification on a single large square tile means the computations foreach pixel are only performed once, instead of once per small squaretile. Thus, the redundancy is reduced by nearly 217-fold. In oneexample, redundancy is not completely eliminated because a slide maycontain several large square tiles, each of which may overlap slightlywith its neighbors.

An upper bound on the redundant calculation percentage can beestablished (slight deviation from this upper bound depends on thenumber of large square tiles needed to cover the tissue and the relativearrangement of these tiles). The redundancy percentage is 1-1/r where ris the redundancy ratio, and r can be calculated as(T/N+1)(sqrt(N)*E+434){circumflex over ( )}2/(sqrt(T)*E+434){circumflexover ( )}2; T is the total number of small square tiles on the slide, Nis the number of small square tiles per large square tile, and E is theedge size of the small square tiles.

FIG. 12A illustrates the layers of an example of the layer structure ofthe architecture 1200. FIG. 12B illustrates example output sizes fordifferent layers and resulting sub-layers of the architecture 1200,showing the tile-resolution FCN configuration. As shown, thetile-resolution FCN configuration included in the tissue classifiermodule 306 has additional layers of 1×1 convolution in a skipconnection, downsampling by a factor of 8 in a skip connection, and aconfidence map layer, and replaces an average pooling layer with aconcatenation layer, and a fully connected FCN layer with a 1×1convolution and Softmax layer. The added layers convert a classificationtask into a classification-segmentation task. This means that instead ofreceiving and classifying a whole image as one tissue class label, theadded layers allow the tile-resolution FCN to classify each small tilein the user-defined grid as a tissue class.

These added and replacement layers convert a CNN to a tile-resolutionFCN without requiring the upsampling performed in the later layers oftraditional pixel-resolution FCNs. Upsampling is a method by which a newversion of an original image can be created with a higher resolutionvalue than the original image. However, upsampling is a time-consuming,computation-intense process, which can be avoided with the presentarchitecture.

There are many methods known in the art for upsampling, includingnearest-neighbor, bilinear, hermite, bell, Mitchell, bicubic, andLanczos resampling. In one example, 2× upsampling means that a pixelwith red green blue (RGB) values will be split into four pixels, and theRGB values for the three new pixels may be selected to match the RGBvalues of the original pixel. In another example, the RGB values for thethree new pixels may be selected as the average of the RGB values fromthe original pixel and the pixels that are adjacent to the neighboringpixel.

Because the RGB values of the new pixels may not accurately reflect thevisible tissue in the original slide that was captured by the digitalslide image, upsampling can introduce errors into the final imageoverlay map produced by the overlay map generator 224.

In an example, instead of labeling individual pixels, thetile-resolution FCN is programmed to analyze a large square tile made ofsmall square tiles, producing a 3D array of values that each representthe probability that one tissue class classification label matches thetissue class depicted in each small tile. A convolution layer performsthe multiplication of at least one input image matrix by at least onefilter matrix. In the first convolution later, the input image matrixhas a value for every pixel in the large square tile input image,representing visual data in that pixel (for example, a value between 0and 255 for each channel of RGB).

The filter matrix may have dimensions selected by the user, and maycontain weight values selected by the user or determined bybackpropagation during CNN model training. In one example, in the firstconvolution layer, the filter matrix dimensions are 7×7 and there are 64filters. The filter matrix may represent visual patterns that candistinguish one tissue class from another.

In an example where RGB values populate the input image matrix, theinput image matrix and the filter matrices will be 3-dimensional (see,FIG. 12C). Each filter matrix is multiplied by each input image matrixto produce a result matrix. All result matrices produced by the filtersin one convolution layer may be stacked to create a 3-dimensional resultmatrix having dimensions such as rows, columns, and depth. The lastdimension, depth, in the 3-D result matrix will have a depth equal tothe number of filter matrices. The resulting matrix from one convolutionlayer becomes the input image matrix for the next convolution layer.

Returning to FIG. 12A, a convolution layer title that includes “/n”,where n is a number, indicates that there is a downsampling (also knownas pooling) of the result matrix produced by that layer. The n indicatesthe factor by which the downsampling occurs. Downsampling by a factor of2 means that a downsampled result matrix with half as many rows and halfas many columns as the original result matrix will be created byreplacing a square of four values in the result matrix by one of thosevalues or a statistic calculated from those values. For example, theminimum, maximum, or average of the values may replace the originalvalues.

The architecture 1200 also adds skip connections (shown in FIG. 12A asblack lines with arrows that connect blue convolution layers directly tothe concatenation layer). The skip connection on the left includesdownsampling by a factor of 8, and the skip connection on the rightincludes two convolution layers that multiply an input image matrix byfilter matrices that each have dimensions of 1×1. Because of the 1×1dimensions of the filter matrices in these layers, only an individualsmall square tile contributes to its corresponding probability vector inthe result matrices created by the purple convolution layers. Theseresult matrices represent a small focus of view.

In all of the other convolution layers, the larger dimensions of thefilter matrices allow the pixels in each medium square tile, includingthe small square tile at the center of the medium square tile, tocontribute to the probability vector in the result matrix thatcorresponds with that small square tile. These result matrices allow thecontextual pixel data patterns surrounding the small square tile toinfluence the probability that each tissue class label applies to thesmall square tile. These result matrices represent a large focus ofview.

The 1×1 convolution layers in the skip connection allow the algorithm toregard the pixel data patterns in the center small square tile as eithermore or less important than pixel data patterns in the rest of thesurrounding medium square tile. The amount of importance is reflected bythe weights that the trained model multiplies by the final result matrixfrom the skip connection layers (shown on the right side of FIG. 12A)compared to the weights that the trained model multiplies by the finalresult matrix from the medium tile convolution layers (shown in thecenter column of FIG. 10A) during the concatenation layer.

The downsampling skip connection shown on the left side of FIG. 12Acreates a result matrix with a depth of 64. The 3×3 convolution layerhaving 512 filter matrices creates a result matrix with a depth of 512.The 1×1 convolution layer having 64 filter matrices creates a resultmatrix with a depth of 64. All three of these results matrices will havethe same number of rows and the same number of columns. Theconcatenation layer concatenates these three results matrices to form afinal result matrix with the same number of rows and the same number ofcolumns as the three concatenated matrices, and a depth of 64+512+64(640). This final result matrix combines the large and small focus ofview matrices.

The final result matrix may be flattened to 2 dimensions by multiplyinga factor by every entry, and summing the products along each depth. Eachfactor may be selected by the user, or may be selected during modeltraining by backpropagation. Flattening will not change the number ofrows and columns of the final results matrix, but will change the depthto 1.

The 1×1 convolution layer receives the final result matrix and filtersit with one or more filter matrices. The 1×1 convolution layer mayinclude one filter matrix associated with each tissue class label in thetrained algorithm. This convolution layer produces a 3-D result matrixthat has a depth equal to the number of tissue class labels. Each depthcorresponds to one filter matrix and along the depth of the resultmatrix there may be a probabilities vector for each small square tile.This 3-D result matrix is the 3-dimensional probability data array, andthe 1×1 convolution layer stores this 3-D probability data array.

A Softmax layer may create a 2-dimensional probability matrix from the3-D probability data array by comparing every value in eachprobabilities vector and selecting the tissue class associated with themaximum value to assign that tissue class to the small square tileassociated with that probabilities vector.

The stored 3-dimensional probability data array or the 2-D probabilitymatrix may then be converted to a tissue class overlay map in the finalconfidence map layer in FIG. 10A, to efficiently assign a tissue classlabel to each tile.

In one example, to counteract shrinkage, input image matrices have addedrows and columns on all four outer edges of the matrices, wherein eachvalue entry in the added rows and columns is a zero. These rows andcolumns are referred to as padding. In this case, the training datainput matrices will have the same number of added rows and columns withvalue entries equal to zero. A difference in the number of padding rowsor columns in the training data input matrices would result in values inthe filter matrices that do not cause the tissue class locator 216 toaccurately label input images.

In the FCN shown in FIG. 12A, 217 total outer rows or columns on eachside of the input image matrix will be lost to shrinkage before the skipconnection, due to the gray and blue layers. Only the pixels located inthe small square tiles will have a corresponding vector in the resultmatrices created by the green layers and beyond.

In one example, each medium square tile is not padded by adding rows andcolumns with value entries of zero around the input image matrix thatcorresponds to each medium square tile because the zeroes would replaceimage data values from neighboring medium square tiles that the tissueclass locator 216 needs to analyze. In this case, the training datainput matrices will not be padded either.

FIG. 12C is a visualization of each depth of an exemplary 3-dimensionalinput image matrix being convoluted by two exemplary 3-dimensionalfilter matrices.

In an example where an input image matrix contains RGB channels for eachmedium square tile, the input image matrix and filter matrices will be3-dimensional. In one of the three dimensions, the input image matrixand each filter matrix will have three depths, one for red channel, onefor green channel, and one for blue channel.

The red channel (first depth) 1202 of the input image matrix ismultiplied by the corresponding first depth of the first filter matrix.The green channel (second depth) 1204 is multiplied in a similarfashion, and so on with the blue channel (third depth) 1206. Then, thered, green, and blue product matrices are summed to create a first depthof the 3-dimensional result matrix. This repeats for each filter matrix,to create an additional depth of the 3-dimensional result matrix thatcorresponds to each filter.

A variety of training sets may be used to train a CNN or FCN model thatis included in the tissue classifier module 306.

In one example, the training set may contain JPEG images of mediumsquare tiles, each having a tissue class label assigned to its centersmall square tile, taken from at least 50 digital images ofhistopathology slides at a resolution of approximately 1 pixel permicron. In one example, a human analyst has outlined and labeled(annotated various tissue classes) all relevant tissue classes orlabeled each small square tile in each histopathology slide asnon-tissue or as a specific type of cells. Classes of various tissue mayinclude tumor, stroma, normal, immune cluster, necrosis,hyperplasia/dysplasia, and red blood cells. In one example, each side ofeach center small square tile is approximately 32 pixels long.

In one example, the training set images are converted to input trainingimage matrices and processed by the tissue classifier module 306 toassign a tissue class label to each tile image of the training image. Ifthe tissue classifier module 306 does not accurately label thevalidation set of training images to match the corresponding annotationsadded by a human analyst, the weights of each layer of the deep learningnetwork may be adjusted automatically by stochastic gradient descentthrough backpropagation until the tissue classifier module 306accurately labels most of the validation set of training images.

In one example, the training data set has multiple classes where eachclass represents a tissue class. That training set will generate aunique model with specific hyperparameters (number of epochs, learningrate, etc.) that can recognize and classify the content in a digitalslide image into different classes. Tissue classes may include tumor,stroma, immune cluster, normal epithelium, necrosis,hyperplasia/dysplasia, and red blood cells. In one example, the modelcan classify an unlimited number of tissue classes, provided each tissueclass has a sufficient training set.

In one example, the training set images are converted into grayscalemasks for annotation where different values (0-255) in the mask imagerepresent different classes.

Each histopathology image can exhibit large degrees of variation invisual features, including tumor appearance, so a training set mayinclude digital slide images that are highly dissimilar to better trainthe model for the variety of slides that it may analyze. Images intraining data may also be subjected to data augmentation (includingrotating, scaling, color jitter, etc.), before being used to train themodel.

A training set may also be specific to a cancer type. In this case, allof the histopathology slides that generated the digital images in aparticular training set contain a tumor sample from the same cancertype. Cancer types may include breast, colorectal, lung, pancreatic,liver, stomach, skin, etc. Each training set may create a unique modelspecific to the cancer type. Each cancer type may also be split intocancer subtypes, known in the art or defined by the user.

In one example, a training set may be derived from histopathology slidepairs. A histopathology slide pair includes two histopathology slidesthat each have one slice of tissue, wherein the two slices of tissuewere located substantially proximal to each other/approximately adjacentin the tumor sample. Therefore, the two slices of tissue aresubstantially similar. One of the slides in the pair is stained with H&Estaining only, and the other slide in the pair is stained with IHCstaining for a specific molecule target. The areas on the H&E stainedslide that correspond to areas where IHC staining appears in the pairedslide are annotated by a human analyst as containing a specific moleculetarget and the tissue class locator receives the annotated H&E slides asa training set. Substantially similar slides include other combinations,such as when the paid includes one with H&E staining and the otherformed with molecular sequencing data scrapped off one of the adjacentslides, or where one is of IHC staining and the other is formed withmolecular sequencing data, or where both are formed of similar molecularsequencing data.

For example, in some embodiments, more than one sample is obtained froma subject—for example, more than one tissue slice can be taken that areadjacent to each other. In some cases, the tissue slices are obtainedsuch that some of the pathology slides prepared from the respectiveslices are imaged, whereas some of the pathology slides are used forobtaining sequencing information.

A suitable training data set may be used for training an optimizationmodel in accordance with embodiments of the present disclosure. In someembodiments, curation of a training data set may involve collecting aseries of pathology reports and associated sequencing information from aplurality of patients. For example, a physician may perform a tumorbiopsy of a patient by removing a small amount of tumor tissue/specimenfrom the patient and sending this specimen to a laboratory. The lab mayprepare slides from the specimen using slide preparation techniques suchas freezing the specimen and slicing layers, setting the specimen inparaffin and slicing layers, smearing the specimen on a slide, or othermethods known to those of ordinary skill. For purposes of the followingdisclosure, a slide and a slice may be used interchangeably. A slidestores a slice of tissue from the specimen and receives a labelidentifying the specimen from which the slice was extracted and thesequence number of the slice from the specimen. Traditionally, apathology slide may be prepared by staining the specimen to revealcellular characteristics (such as cell nuclei, lymphocytes, stroma,epithelium, or other cells in whole or part). The pathology slideselected for staining is traditionally the terminal slide of thespecimen block. Specimen slicing proceeds with a series of initialslides that may be prepared for staining and diagnostic purposes. Aseries of the next sequential slices may be used for sequencing, andthen final, terminal slides may be processed for additional staining. Ina case when the terminal, stained slide is too far removed from thesequenced slides, another slide may be stained which is closer to thesequenced slides such that sequencing slides are broken up by stainingslides. While there are slight deviations from slice to slice, thedeviation is expected to be minimal as the tissue is sliced atthicknesses approaching 4 um for paraffin slides and 35 um for frozenslides. Laboratories generally confirm that the distance, usually lessthan 40 um (approximately 10 slides/slices), has not produced asubstantial deviation in the tissue slices.

In (less frequent) cases where slices of the specimen vary greatly fromslice to slice, outliers may be discarded and not further processed. Thepathology slides 510 may be varying stained slides taken from tumorsamples from patients. Some slides and sequencing data may be taken fromthe same specimen to ensure data robustness, while other slides andsequencing data may be taken from respective unique specimens. Thelarger the number of tumor samples in the dataset, the more accuracy canbe expected from the predictions of cell-type RNA profiles. In someembodiments, a stained tumor slide may be reviewed by a pathologist foridentification of cellular features, such as the quantity of cells andtheir differences from the normal cells of that or similar type.

In this case, the trained tissue classification model 320 receivesdigital images of H&E stained tissue to predict tiles that may containIHC staining or a given molecule target and the overlay map generator326 produces an overlay map showing which tiles are likely to containthe IHC target or given molecule. In one example, the resolution of theoverlay is at the level of an individual cell.

The overlay produced by a model trained by one or more training sets maybe reviewed by a human analyst in order to annotate the digital slideimage to add it to one of the training sets.

The pixel data patterns that the algorithm detects may representvisually detectable features. Some examples of those visually detectablefeatures may include color, texture, cell size, shape, and spatialorganization.

For example, color on a slide provides contextual information. Forexample, an area on the slide that is purple may have a higher densityof cells and may be more likely to be invasive tumor. Tumor also causesthe surrounding stroma to become more fibrous in a desmoplasticreaction, which causes normally pink stroma to appear blue-grey. Colorintensity also helps to identify individual cells of a certain type (forexample, lymphocytes are uniformly very dark blue).

Texture refers to the distribution of stain within cells. Most tumorcells have a rough, heterogeneous appearance, with light pockets anddark nucleoli within their nuclei. A zoomed-out field of view with manytumor cells will have this rough appearance. Many non-tumor tissueclasses each have distinguishing features. Furthermore, patterns oftissue classes that are present in a region can indicate the type oftissue or cell structures present in that region.

Additionally, cell size often indicates tissue class. If a cell isseveral times larger than normal cells elsewhere on the slide, theprobability is high that it is a tumor cell.

The shape of individual cells, specifically how circular they are, canindicate what type of cell they are. Fibroblasts (stromal cells) arenormally elongated and slim, while lymphocytes are very round. Tumorcells can be more irregularly shaped.

The organization of a group of cells can also indicate tissue class.Frequently, normal cells are organized in a structured and recognizablepattern, but tumor cells grow in denser, disorganized clusters. Eachtype and subtype of cancer can produce tumors with specific growthpatterns, which include cell location relative to tissue features, thespacing of tumor cells relative to each other, formation of geometricelements, etc.

The techniques herein may be extended to other architectures. FIG. 14,for example, illustrates an imaging-based biomarker prediction system1400 that similarly employs separate pipelines for tissue classificationand cell classification. The system 1400 may be used for variousbiomarker determinations, including PD-L1 as described in the examplesherein. Further, the system 1400, like other architectures herein may beconfigured to predict biomarker status and tumor status and tumorstatistics based on 3D image analysis.

The system 1400 receives one or more digital images of a histopathologyslide and creates a high-density, grid-based digital overlay map thatidentifies the majority class of tissue visible within each grid tile inthe digital image. The system 1400 may also generate a digital overlaydrawing identifying each cell in a histopathology image, at theresolution level of an individual pixel.

The system 1400 includes a tissue detector 1402 that detects the areasof a digital image that have tissue and stores data that includes thelocations of the areas detected to have tissue (e.g., pixel locationsusing a reference location in an image, such as a 0,0 pixel location).The tissue detector 1402 transfers tissue area location data 1403 to atissue class tile grid projector 1404 and to a cell tile grid projector1406. The tissue class tile grid projector 1404 receives the tissue arealocation data 1403, and, for each of several tissue class labels,performs a tissue classification on a tile. A tissue class locator 1408receives resulting tile classification and calculates a percentage thatrepresents the likelihood that the tissue class label accuratelydescribes the image within each tile to determine where each tissueclass is located in the digital image. For each tile, the total of allof the percentages calculated for all tissue class labels will sum to 1,which reflects 100%. In one example, the tissue class locator 1408assigns one tissue class label to each tile to determine where eachtissue class is located in the digital image. The tissue class locatorstores the calculated percentages and assigned tissue class labelsassociated with each tile.

In an example, the system 1400 includes a multi-tile algorithm thatconcurrently analyzes many tiles in an image, both individually and inconjunction with the portion of the image that surrounds each tile. Themulti-tile algorithm may achieve a multiscale, multiresolution analysisthat captures both the contents of the individual tile and the contextof the portion of the image that surrounds the tile. Because theportions of the image that surround two neighboring tiles overlap,analyzing many tiles and their surroundings concurrently instead ofseparately analyzing each tile with its surroundings reducescomputational redundancy and results in greater processing efficiency.

In an example, the system 1400 may store the analysis results in a3-dimensional probability data array, which contains one 1-dimensionaldata vector for each analyzed tile. In one example, each data vectorcontains a list of percentages that sum to 100%, each indicating theprobability that each grid tile contains one of the tissue classesanalyzed. The position of each data vector in the orthogonal2-dimensional plane of the data array, with respect to the othervectors, corresponds with the position of the tile associated with thatdata vector in the digital image, with respect to the other tiles.

The cell type tile grid projector 1406 receives the tissue area locationdata 1403 and identifies and classifies cells in a tile and projects acell type tile grid onto the areas of an image with tissue. A cell typelocator 1410 may detect each biological cell in the digital image withineach grid, prepare an outline on the outer edge of each cell, andclassify each cell by cell type. The cell type locator 1410 stores dataincluding the location of each cell and each pixel that contains a cellouter edge, and the cell type label assigned to each cell.

The overlay map generator and metric calculator 1412 may retrieve thestored 3-dimensional probability data array from the tissue classlocator 1408, and convert it into an overlay map that displays theassigned tissue class label for each tile. The assigned tissue class foreach tile may be displayed as a transparent color that is unique foreach tissue class. In one example, the tissue class overlay map displaysthe probabilities for each grid tile for the tissue class selected bythe user. The overlay map generator and metric calculator 1412 alsoretrieves the stored cell location and type data from the cell typelocator 1410, and calculates metrics related to the number of cells inthe entire image or in the tiles assigned to a specific tissue class.

FIG. 15A illustrates an overview of an example process 1500 implementedby the imaging-based biomarker prediction system 1400, showing modelinference pipeline for predicting a biomarker, in this example PD-L1.The process 1500 takes advantage of the fully convolutional modelarchitecture for the system 1400 to process many tiles in parallel. Inan example, the process 1500 was able to take 2.8 seconds to classify asingle 4096×4096 pixel image, using a GeForce GTX 1080 Ti GPU and 6thGeneration Intel® Core™ i7 processor. The process 1500 further includedtissue detection and artifact removal algorithms so as to function in afully automated manner in a real-world setting where slides may includeartifacts.

At a first process 1502, initial tissue segmentation is performed by thetissue detector 1402, for example, applying a tissue masking algorithmto automatically contour the tissue (red outline) in order to produce abounding box (not shown) around the tissue of interest. Aligning to theupper left corner of the bounding box, the tissue region is divided intolarge non-overlapping 4096×4096 input windows (blue dashed lines).Typically, between 10-30 input windows are needed to cover the tissue.Any large window area extending beyond the bounded region is padded with0s (gray region).

A process 1504 performs trained classification model prediction. In theillustrated example, large input windows contained 128×128=16,384 small32×32 tiles (grids are much finer than shown). Large input windows werepadded by 0s on all sides (length 217) to account for edges ofoverlapping 466×466 tiles centered on each 32×32 small tile. Each largewindow was passed through one or more trained model 1506 of the deeplearning framework (including the tissue classification process ofprojector 1404 and the cell classification process of projector 1406).With the trained model(s) 1506 being fully convolutional, in an example,each tile within the large input window is processed in parallel,producing a 128×128×3 probability cube (there are 3 classes). Each 1×1×3vector of this probability cube corresponds to a 32×32 px region at thecenter of each 466×466 tile in the original image. The resultingprobability cubes are assembled into a probability map of the wholeimage.

A process 1508 displays images associated with a tissue masking step ofprocess 1502. An assembled probability map generated by the process 1504is passed through this tissue mask to remove background. In the example,both background and marker area are removed by the masking algorithm ofthe process 1508.

A process 1510 displays images showing a classification map thatidentifies one or more classified regions, such as each the differentregions for each of the biomarker classifications. In an example, themaximum probability class (argmax) is assigned to each tile through theprocess 1508, producing a classification map of the three biomarkerclasses (PD-L1+, PD-L1-, Other), at the process 1510. The classificationmap illustrates each of the these biomarker classifications and theiridentified location corresponding to the original histopathology image.

A process 1512 performs statistical analysis on biomarkerclassifications from the classification map and displays a resultingprediction score for the biomarker. In this example, the number ofpredicted PD-L1 positive tiles is divided by the total number ofpredicted tumor tiles to achieve an exemplary model score.

The tissue class locator 1408 may have an architecture like that ofarchitecture 1200, for example. The architecture is similar to that ofthe FCN-tile resolution classifier. The architecture 1550 may be formedof three major components: 1) a fully convolutional residual neuralnetwork (e.g., built on ResNet-18) backbone that processes a large fieldof view image (FOV), 2) two branches that process small FOVs, and 3)concatenation of small and large FOV features for multi-FOVclassification. The ResNet-18 backbone contains multiple shortcutconnections, indicated by dotted lines, where feature maps are alsodownsampled by 2. The small FOV branches emerge after the secondconvolutional block. The feature maps of the small FOV branches aredownsampled by 8 to match the dimensions of the ResNet-18 feature map.These feature maps are concatenated before passing through a softmaxoutput to produce a PD-L1 biomarker prediction (confidence) map.

In an example, the backbone of the model includes an 18-layer version ofResNet (ResNet-18) with some modifications. The ResNet-18 backbone wasconverted into a fully convolutional network (FCN) by removing theglobal average pooling layer and eliminating zero-padding in downsampledlayers. This enabled the output of a 2D probability map rather than a 1Dprobability vector (see, FIG. 15B). In the illustrated example, the tilesize (466×466 pixels) was over twice the tile size of a standard ResNet,providing our model with a larger FOV that allows it to learnsurrounding morphological features. Although the tissue class locator1408 in this example adds multiple field of view (multi-FOV)capabilities to a ResNet architecture, it should be understood thattissue class locator 1408 could be comprised of a distinct networkarchitecture that has been adapted to incorporate the multi-FOV approachdisclosed here.

The FCN configuration of the architecture 1505 provides numerousadvantages, including overcoming accuracy degradation challengestraditionally suffered by “very deep” neural networks (including neuralnetworks with more than 16 convolutional layers) (see, e.g., He et al.,Deep Residual Learning for Image Recognition, (2015) (arXivID:1512.03385v1, and Simonyan et al., Very Deep Convolution Networks forLarge-Scale Image Recognition (2014), arXiv ID:1409.1556v6). Thearchitecture 1550 includes a stack of convolutional layers interleavedwith “shortcut connections,” which skip intermediate layers. Theseshortcut connections use earlier layers as a reference point to guidedeeper layers to learn the residual between layer outputs rather thanlearning an identity mapping between layers. This innovation improvesconvergence speed and stability during training, and allows deepernetworks to perform better than their shallower counterparts.

The tissue class locator 1408 may include two additional branches withreceptive fields restricted to a small FOV (32×32 pixels) in the centerof the second convolutional feature map (see, FIG. 15B). One branchpasses a copy of the small FOV through a convolutional filter, while theother branch is a standard shortcut connection with downsampling. Thefeatures produced by these additional branches are concatenated to thefeatures from the main backbone just before the model outputs areconverted into probabilities, in a softmax layer. In this way, thetissue class locator 1408 combines information from multiple FOVs, muchlike a pathologist relies on various zoom levels when diagnosing slides.Moreover, the architecture ensures that the central region of each tilecontributes more to classification than the tile edges, resulting in amore accurate classification map across the entire histopathology image.

FIG. 15B illustrates an example training process 1570 for theimaging-based biomarker prediction system 1400 and generation of anoverlay map output, predicting the location of a PD-L1 biomarker fromanalysis of IHC and H&E histopathology images. At a model trainingprocess 1570, matching areas on IHC and H&E digital images wereannotated by a medical professional. In some examples, however, stainson adjacent tissue slices may be automatically annotated. Images wereannotated showing PD-L1+ and PD-L1- and fed to the system for training.The annotated regions of the H&E image were tiled into overlapping tiles(466×466 pixels) with a stride of 32 pixels, producing training data.The tissue class locator 1408 was then trained using a cross entropyloss function. The yellow square in the model schematic depicts thecentral region that is cropped for the small FOVs. The resulting PD-L1classification model is stored in trained deep learning framework 1574.

FIG. 15B also illustrates an example prediction process 1572, where eachimage was divided into large non-overlapping 4096×4096 input windows(blue dashed lines). Each large window was passed through the trainedmodel. Because the deep learning framework 1574 is fully convolutional,each tile within the large input window was processed in parallel,producing a 128×128×3 probability cube (the last dimension representsthree classes). The resulting probability cubes were slotted into placeand assembled to generate a probability map of the whole image. Theclass with the maximum probability was assigned to each tile and a PD-L1prediction is report is generated.

FIGS. 16A-16F illustrate input histopathology images received by theimaging-based biomarker prediction system 1400, corresponding overlaymaps generated by the system 1400 to predict the location of IHC PD-L1biomarker, and corresponding IHC stained tissue images used asreferences to determine the accuracy of overlay maps. The IHC stainedtissue images were obtained from a test cohort but were not applied tothe system 1200 during model training. FIGS. 16A-16C illustrate arepresentative PD-L1 positive biomarker classification example. FIG. 16Adisplays an input H&E image; FIG. 16B displays a probability mapoverlaid on the H&E image: FIG. 16C displays a PD-L1 IHC stain forreference. FIGS. 16D-16F illustrate a representative PD-L1 negativebiomarker classification example. FIG. 16D displays an input H&E image;FIG. 16E displays a probability map overlaid on the H&E image; and FIG.16F displays a PD-L1 IHC stain for reference. The color bar indicatesthe predicted probability of the tumor PD-L1+ class.

Among the advantages offered by the deep learning frameworks herein,they can exhibit improved accuracy by disrupting shift invariance. Shiftinvariance, or homogeneity, is a property of linear filters such asconvolution, where the response of the filter does not explicitly dependon the position. In other words, if we shift a signal, the output imageis the same but with the shift applied. While shift invariance isdesirable in most image classification tasks (Le Cun, 1989), in examplesherein we typically do not want objects close to the edge of the tile tocontribute equally to the classification.

FIG. 17 illustrates an example advantage of a multi-FOV strategy such asthat described in reference to FIGS. 14, 15A, and 15B. In a top portion,large FOV (red box) contains both PD-L1+ tumor cells (purple, upperleft) and stroma (pink). Only stroma falls within the small FOV (greenbox). When passed through a convolutional layer of the tissue classlocator 1408, the tumor area produces a unique pattern (colored squares)distinct from the pattern produced by stromal areas (white squares).After the patterns from large and small FOV branches are concatenated,the model is likely to predict “Other”. In the bottom portion, the fieldof view has shifted, and now the PD-L1+ tumor area falls within thesmall FOV. This tumor area will produce the same convolutional filterpatterns in both large and small FOV branches (colored squares). Uponconcatenating the learned features, the tissue class locator 1408 is nowmore likely to predict PD-L1+ tumor. Thus, without the small FOV used intraining the tissue class locator 1408, the system 1400 may havepredicted PD-L1+ tumor for both images. Instead, the multi-FOV strategyof the architecture in FIG. 15 allows the network to take advantage ofrich contextual information in the surrounding area, while stillfavoring what is in the center of the image for classification.

Yet other architectures may be used for any of the classifier examplesherein to predict biomarker status, tumor status, and/or metricsthereof, in particular using multiple instance learning techniques.

In examples discussed herein, classification model architectures, e.g.,based on FCN architectures described in FIG. 12A, were trained withdigital images of a histopathology slide, which may include a matrix ofannotations. Training from such digital images is performed on a tile bytile basis, where for example only tiles that have an annotation (i.e.,label) are supplied as training tiles to the deep learning framework.Tiles of the digital image without annotations may be discarded.Further, each column and row of a matrix corresponds to a distinct gridhaving N×M pixels of the digital image. In order to properly assign theannotations from the matrix having columns and rows to a digital imagehaving a plurality of tiles, it may be advantageous to, in someexamples, take the column (i) and row (j) from the matrix and assign theannotation to the center region of a grid starting at pixel N(i) andM(j) and extending to the next [N−1] to [M−1] pixels, where the tilewith central region extending across that range is assigned theannotation of the matrix at i,j. Thus, the matrix may accuratelyrepresent annotations at a tile by tile basis within the larger digitalimage.

The FCN architectures may take a large tile as input while the labelcomes from the central region mapping to the annotation mask points. FCNarchitectures may learn from both the central region of the large tileand the pixels surrounding the central region, where the central regioncontributes more to the prediction. Additionally, slide metadata may bestored in a feature vector, such as a vector associated with the slidewhich identified slide level labeling, including the patient featureswhich may improve model performance. A plurality of tiles from grids ofdigital images and corresponding annotation matrices are sequentiallyprovided to the FCN architecture, to train the FCN architecture inclassifying tiles of N×M size according to the annotations included inthe matrix and the slides themselves according to the annotationincluded in the feature vector. An output of the FCN architecture mayinclude a matrix having predicted classifications on a tile-by-tilebasis and can be aggregated to a vector having predicted classificationson a slide by slide basis. The matrix may be converted to a digitaloverlay by associating the highest classifications of each tile to acolor which may be superimposed over the corresponding grid location inthe digital image. In some examples, the matrix may be converted tomultiple digital overlays, wherein each overlay corresponds to eachclassification and an intensity of an associated color is assigned tothe overlay based upon the percentage of confidence ratio associatedwith the respective classification. For example, a tile having a 30%likelihood as tumor, 50% likelihood as stroma, and a 20% likelihood asnormal may be assigned a single overlay for stroma, as the highestlikelihood of tissue in the tile, or may be assigned a first overlaywith a 30% intensity of a first color and a second overlay with a 50%intensity of a second color, to identify the type of tissue that thetile may comprise.

However, even a classification model based purely off of an architecturewhich may not support tile-by-tile annotations, such as an architecturesimilar to Resnet-34 or Inception-v3, may be trained with a digitalimage of a pathology slide with only a vector of annotations, where eachentry of the vector is an annotation of a patient feature or metadatawhich applies to the slide. In some examples, even an architecture thatsupports tile-by-tile annotations may not have access to tile-by-tileannotations for a specific feature.

To use histopathology images in training neural networks images that donot have tile annotation or to train neural networks to identifybiomarkers trained on molecular training data, in some examples, thepresent techniques include deep learning training architecturesconfigured for label-free training. In examples, architectures do notrequire tile level annotation when training tissue classificationmodels. Moreover, the label-free training architectures are neuralnetwork agnostic, in that they allow for label-free training that isagnostic to the configuration of the neural network (i.e., ResNet-34,FCN, Inception-v3, UNet, etc.). The architectures are able to analyze aset of possible training images and predict which tiles to exclude fortraining. Therefore, images may be discarded from training in someexamples, while in other examples tiles may be discarded but the rest ofthe image may be used for training. These techniques result in much lesstraining data which greatly reduces the time required to train, and insome instances update training of, classification models herein.Further, the techniques do not require pathologist labeling, whichgreatly reduces the time it takes to train a classification model andwhich avoids annotation errors and annotation variability acrossprofessionals.

Instead, in some examples, training may be performed with weaklysupervised learning that involves only image level labeling, and nolocal labeling of tissue, cell, tumor, etc. The architectures may beconfigured with a label-less training front end having an algorithm withcustomized cost function that chooses which tile(s) should be used asthe input with specific label. The process may be iterative, first,treating each histopathology image as a collection of tiles, where asingle label of the image is applied to all the tiles in the collection.The tiles may be applied to an inference pipeline, such as through anetwork like ResNet 34, Inception-v3, or FCN, and predefined tileselection criteria such as probabilities of the neural network outputmay be used to select which output image tiles will be provided as aninput to the same neural network for next round. This process may berepeated many times, given enough collections and tiles as input to theneural network, it will learn to differentiate tiles with differentclasses with higher accuracy as more iterations are performed.

FIG. 18 illustrates an example machine learning architecture 1800 in anexample configuration for performing label-free annotation training of adeep learning framework, to execute processes described in examplesherein. A deep learning framework 1802, which may be similar to otherdeep learning frameworks described herein having multiscale andsingle-scale classification modules, includes a pre-processing &post-processing controller 1804, performing processes are described insimilar examples in FIGS. 1 and 3. The deep learning framework 1802includes a cell segmentation modules 1806 and a tissue classifier module1808, each of which being configured as a tile-based neural networkclassifier. The deep learning framework 1802 further includes a numberof different biomarker classification models 1810, 1812, 1814, and 1816,each of which may be configured to have a different neural networkarchitecture, where some may have a multiscale configuration and somemay have a single-scale configuration. These different neural networkarchitectures may be configured for training using annotated ornon-annotated images. Some of these architectures may be configured fortraining using tile-annotated training images, while others of thesearchitectures are configured for training using training images with notile-annotations. For example, some architectures may be configured toaccept only slide-level annotations on images (i.e., an annotation foran entire image and not an annotation identifying particular tilecharacteristics, cell segmentations, or tissue segmentations). Exampleneural network architecture types for the modules 1810-1816, include,ResNet-34, FCN, Inception-v3, and U Net.

Annotated images 1818 may be provided to the deep learning framework1802 for training of the various classification modules, usingtechniques described above. In some examples, the entire histopathologyimage is provided to the framework 1802 for training. In some examples,the annotated images 1818 are passed directly to the deep learningframework 1802. In some examples, the annotated images 1818 may beannotated at a granularity that is to be reduced. As such, in someexamples, a multiple instance learning (MIL) controller 1821 may beconfigured to further separate the annotated images 1818 into aplurality of tile images each corresponding to a different portion ofthe digital image, and the MIL controller 1821 applies those tile imagesto the deep learning framework 1802. With the architecture 1800,however, non-annotated images 1820 may be used for classification moduletraining, by first providing those images 1820 to the MIL controller1821 having a front end tile selection controller 1822. In someexamples, the MIL controller 1821 may be configured to separatenon-annotated images 1820 into a plurality of tile images eachcorresponding to a different portion of the digital image, and the MILcontroller 1821 applies those tile images to the deep learning framework1802. In an example, the architecture 1800 deploys weakly supervisedlearning to train convolutional neural network architectures (FCN,ResNet 34, Inception-v3, etc.) to classify local tissue regions usingonly slide-level labels. Weakly supervised learning does not requirelocalized annotations, so labeling can be done faster, resulting in alarger set of labeled slides. This architecture 1800 thus can be used totrain models that supplement or improve on FCN classifications, or evento train the FCN-based model itself.

In this illustrated example, the front end tile section controller 1822is a configured in a feedback configuration that allows for combining atile section process with a classification model, allowing the tilesection process to be informed by a neural network architecture such asan FCN architecture. In some examples, the tile selection processperformed by the controller 1822 is a trained MIL process. For example,one of the biomarker classifications models 1810-1816 may generate anoutput, during training, where that output is used as an initial inputto guide the tile selection controller 1822. With a MIL process being aniterative process where usually the initial tile selection ischallenging, by informing the MIL process of the controller 1822 withguidance from, for example, an FCN architecture prediction, the MILprocess of the controller 1822 will start with better examples andconverge much faster to a stable and useful FCN classifier. In yetanother example, combining the results from an FCN (or other neuralnetwork) architecture and the MIL process of controller 1822 may includeonly combining the results for the vector outputs in a concatenationlayer so that the matrix output is by voting for the best. FCNarchitecture and MIL process of the controller 1822 can be configuredthe same prediction task, i.e., looking for the same biomarker. However,in some cases, the two classification process may have differentprediction outputs; and in those cases, combining the result by votingfor the best may be performed. In yet another example, the outputs fromMIL framework and FCN architecture can be combined to get better slidelevel prediction, in which MIL is using slide level loss function aslearning criteria, and output from the FCN architecture is used to comeup with guided truth for MIL loss calculation.

The tile section controller 1822 may be implemented in a number ofdifferent ways.

An example tile selection process for the controller 1822 is explainedin reference to a basic framework for a single class. In the singleclass example, a tile of histopathology image is classified as eitherbeing in the target class (class 1) or not (class 0). Class 0 can bethought of as background, anything that's not in our target class. In aninstance-based MIL process, tiles need to be selected that should beused as examples used in training. In the case of a single classproblem, the controller 1822 may be configured with a trained model thatwould return the following, classifications: if a slide does not haveany tiles that belong to the target class, all tiles should return a lowinference score of zero. During training, that slide would be labeled asbeing class 0; and if a slide has any tiles that belong to the targetclass, it will be labeled as class 1. Those tiles that belong to thetarget class should return an high inference score of 1, and all othertiles should return a low inference score of 0. In order to train theclassifier model (e.g., models 1810-1816), tiles need to be identifiedthat are representative of the slide-level class. Since we know that aclass 0 slide has no tiles that are of class 1, then any tile can beused as a training example. However, the best selection would be to usetiles that the model is performing worst with. Since all tiles shouldhave an inference score of 0, then the tiles with the highest scoresshould be used to train the model. These tiles with the highest scoresare called the “top k” tiles, where k is an integer indicating how manytiles are being used for training from that slide (e.g. 5, 10, 15).

For class 1 slides, the controller 1822 may identify tiles that are mostlikely to be of class 1. This determination may be complicated by thefact that a class 1 slide can contain tiles from both class 0 andclass 1. However, with the classifier model trained with class 0 tilesfrom the class 0 slides, this means that any tiles in the class 1 slidethat are similar to the class 0 tiles should have lower inferencescores. Similarly, this means that tiles that are not similar to theclass 0 tiles should have higher scores. Thus, the tiles that are mostlikely to really be class 1 are those tiles with the highest inferencescores. Thus, the top k tiles should again be selected as examples fortraining the model.

This means that for both class 0 and class 1 slides, the top k scoredtiles should be used as training examples as seen in tile selectionframework 1900 in FIG. 19. The row of numbers represent the inferencescores for the different tiles. Initial a non-tile annotatedhistopathology image is provided at 1902 and a model inference isperformed on each of the tiles in the image a process 1904.

The framework 1900 may be used to calculate class prediction scores onall tiles in all slides by running the model inference 1904, and thetiles with the highest scores from each slide (the top k tiles, labeled1906) are selected for training the model. The tiles 1906 are given thesame label as the slide-level label received at 1902 (e.g. tiles from aclass 0 slide will be given a class 0 label). After selecting tiles fromall slides, a model is trained at a process 1908 in a single epoch (or,iteration).

At the conclusion of the training epoch (the tiles were used once forupdating the model weights), the framework 1900 is then used again tocalculate new prediction scores for all the tiles in all the slides. Thenew prediction scores are used to identify new tiles to use fortraining. This process of tile scoring and selecting the top k tiles fortraining is repeated until the model reaches some criteria for stopping(e.g. convergence of performance on a withheld set of slides used forvalidation), as determined by a process 1910.

There are several advantages of using a weakly supervised learningconfiguration like that of 1800 in FIG. 18. Strongly supervisedannotations and local annotations, where a pathologist manually marksexamples of tissue classes throughout an entire image, are costly andinefficient, wherein the architecture 1800 can train a model with asingle label on datasets that have orders of magnitude more slides.Further, it is possible to have classification targets for whichannotations cannot be done. For example, there are genetic mutations,currently found through genotype biomarkers, that may be correlated withtissue features, but what those tissue features are heretofore unknown.But with the present techniques, the genotype biomarker can now be usedas the slide-level label and the training framework of the architecture1800 can be used to identify the tissue morphologies that can be used topredict what genotypes are present. Whereas, conventional RNA/DNAanalysis to predict genotyping can takes weeks, image classification topredict genotypes using the present techniques can be done in hours andwith much larger training sets.

To avoid overfitting situations, in some examples the tile selectioncontroller 1822 may be configured to perform randomized tile selection.For example, for training with small datasets (<300 slide images), aframework like framework 1900 of FIG. 19, may overfit to a few tiles inthe class 1 slides. This situation may occur, because if a class 1 tileis used for training, its score will be higher in the next epoch.Therefore, the class 1 tile will likely be selected for training againin the next epoch, further increasing its inference score, andincreasing its probability of being selected again, and so on.

FIG. 20 illustrates a framework 2000 that may be used to avoidoverfitting. The framework 2000 is similar to framework 1900, andbearing similar reference numerals, but with a random tile selector 2012is used to randomly select from among high score tiles and then sendsthe randomly selected tiles to a training process 2008. The model 2004is still used to determine the inference scores, and tiles are selectedbased on their scores. However, if a tile's inference score is high, itcould still be considered likely to be a class 1 tile even if it's notone of the top k tiles. For example, the framework 2000 may set a lowerthreshold score of 0.9 (or any value), then any tile with a score of 0.9or higher might be used as a training example. That is, any of the tiles2006 could be used for training because their scores are above adetermined threshold (e.g., a threshold of 0.9). The random high scoretile selection 2012 then randomly determines which of these tiles areprovided to the model training process 2008. In other examples, thetiles sent to the random high score tile sector 2012 are the top ktiles. Further, in some examples, tile selection probabilities appliedby the selector 2012 may be fully random, across all tile scores, whilein other examples, the selection probabilities may be partially random,in that tiles with certain scores or within certain score ranges havedifferent random selection probabilities tan tiles with other scores orwithin other score ranges.

FIG. 21 illustrates another framework 2100 for addressing overfitsituations. In some examples, when training with smaller datasets (<300slide images), the framework of FIG. 19 could overfit slide images bypredicting all tiles in class 0 slides as class 0, and all tiles inclass 1 slides as class 1 (even though not all tiles in class 1 slidesare really of class 1). This means it is possible to misclassify thetiles in the class 1 slides. With the framework 2100, however, randomtile selection is performed on high score tiles 2101, as with theframework 2000, but additionally, random tile selection is performed onlow score tiles 2103. In the illustrated example, a random low scoretile selector 2102 feeds a class 0 model training process 2104. A randomhigh score tile selector 2106 fees a class 1 model training process2108.

The examples of FIGS. 19-26 are discussed in the context of single classtraining. The present techniques of label-free training may be used formulti-class training, as well. For a multi-class problem, it is possibleto have a set of slide-level labels such that no slides have a class 0label. For example, with colorectal cancer (CRC) when training models topredict the consensus molecular subtype (CMS), the CMS class is used toguide targeted treatment, but only using genotype biomarkers, i.e.,mutations in RNA data. The present techniques however predict genotypesthrough imaging, which allows the targeted treatment to be started inhours and testing a patient, rather than having to wait weeks to do anRNA analysis. While examples are described in reference CMS, classifiermodules for other image-based biomarkers herein may be trained with thearchitecture 1800, including, PD-L1, TMB, etc.

One issue with multi-class training is that not all images will containall classes. For CMS, for example, the available labels for trainingimages could be CMS1, CMS2, CMS3, or CMS4. However, not all tiles ineach image would necessarily contain biomarkers that are indicative ofany of these four classes. This can create an situation where there areno class 0 slides that can be used to identify class 0 tiles, and thatcan result in the a trained model misclassifying tissue types that haveno predictive value, resulting in a lower accuracy model. Furthermore,it is possible that a slide could contain features that arerepresentative of other classes than the slide-level label. For examplewith CMS, more than one subtype could be present in the CRC. This meansthat while a specific image could be given a CMS1 label because that isthe dominant subtype for that sample, the whole slide image couldcontain some CMS2 tissue.

To achieve multi-class training, in some examples, the tile selectioncontroller 1822, in FIG. 18, is configured to perform a series ofprocesses to identify tissue features that correlate with the differentclass subtypes, for example, four CMS subtypes. First, the tileselection controller 1822 may be trained by identifying only positivein-class examples. Second, the tile selection controller 1822 may applymodel training by identifying positive in-class tiles and negativeout-of-class tiles by low positive in-class scores. Third, the tileselection controller 1822 may apply model training by identifyingpositive in-class tiles and negative out-of-class tiles by high negativescores. Examples of each process are now described in reference totraining a CMS biomarker classifier, as depicted in FIGS. 22-26.

FIG. 22 illustrates a framework 2200 that may be used to identify whichCMS class each tissue feature is most correlated with, whether thattissue feature is relevant or not in classification. FIG. 22 illustratesan example of the first process identifying only positive in-classexamples, and showing only two classes for simplicity purposes. For eachtile in a histopathology image, a list of possible scores is given, witheach row corresponding to the class shown on the left (0, 1, and 2). Asshown, for the class 1 slide image (left side of FIG. 22), the tileswith high class 1 scores (shaded) are used for training, and for theclass 2 slide image (right side of FIG. 22), the tiles with high class 2scores (shaded) are used for training.

In this example, the result will be that all tiles are classified aseither 1 or 2, and the probability of being classified as class 0 is0.00. FIG. 23 illustrates a resulting overlay map showingclassifications for the CMS biomarker with four classes. The tiles arecolored coded based on the inference scores for the four CMS classes,with CMS1 (microsatellite instability immune) shown in red, CMS2(epithelial gene expression profile, WNT and MYC signalling activation)shown in green, CMS3 (epithelial profile with evident metabolicdysregulation) shown in dark blue, and CMS4 (mesenchymal, prominenttransforming growth factor-(3 activation) shown in light blue. In theillustrated example, the transparency of the tiles has been adjusted toindicate the inference score, with higher scores being more opaque andlower scores being more transparent.

FIG. 24 illustrates the framework 2200 applied in the second process,i.e., where the model training continues by identifying positivein-class tiles and negative out-of-class tiles by low positive in-classscores. With the first process in FIG. 22 classifying all tiles as oneof the non-zero classes, the second process in FIG. 24 identifies tilesthat are more likely to be for the background class 0. In theillustrated example, the process does this by identifying tiles thathave scores below a threshold for the slide-level class. If a tile inthe class 1 slide image has a score <0.1, it is marked low score as apossible slide image to use as a class 0 example. A similar process isperformed for the class 2 slide image. FIG. 25 illustrates a resultingoverlay map showing classifications of CMS. If a tile is predicted to beclass 0, it is made transparent. In comparison to FIG. 23, in FIG. 25,this second process identifies some tissue types as being class 0, whichmeans that an image would be background tissue that is uncorrelated withany of the CMS classes. At the same time, the second process is able toidentify the different tissue types or tissue features that arecorrelated with each of the four classes. The tiles are colored based onthe inference scores for the four CMS classes as in FIG. 23, but if atile does not show any coloring, it is predicted to be a class 0 tilethat has no class predictive value.

FIG. 26 illustrates the framework 2200 applied in the third process,i.e., continuing model training by identifying positive in-class tilesand negative out-of-class tiles by high negative scores. As noted above,for the CMS biomarker, the CMS classes in training images are notmutually exclusive. It is possible to have CMS2 tissue types or tissuefeatures in a CMS1 slide image. Therefore, in some examples, if the tileselection controller 1822 continues labeling tiles as class 0 due tohaving a low score in the slide-level class, some tiles could bemislabeled. From the second process above, class 0 tissue that has lowor no correlation with any of the CMS class has already been identified.There are tiles that have a high class 0 score. Therefore, in the thirdprocess, illustrated in FIG. 26, the class 0 tiles can be identifiedbased on having a high class 0 score.

Returning to FIG. 18, the architecture 1800 may be used to train any ofthe biomarker classification models 1810-1816 to classifier a differentbiomarker. CMS is discussed in reference to FIGS. 22-26 by way ofexample. Further, the architecture 1800 is agnostic to the convolutionneural network configuration, i.e., each module 1810-1816 may have thesame or different configurations. In addition to the FCN architecture ofFIGS. 10A-10C, the modules 1810-1816 may be configured with a ResNetarchitecture, such as that shown in FIG. 27. The ResNet architectureprovides skip connections that help avoid the vanishing gradientsproblem during training, as well as the degradation problem for largerarchitectures. Pretrained ResNet models of different sizes are availablefor initializing training of new models, including but not limited toResNet-18 and ResNet-34. In addition to ResNet, the architecture 1800may be used to trained neural networks with convolution layers creatinga feature map that is input to fully connected classification layers,such as AlexNet or VGG; networks with layered modules that simultaneousapplication of multiple convolutional kernels, such as Inception v3;networks designed to require fewer parameters for increased processingspeed, such as MobileNet, SqueezeNet, or MNASNet; or customizedarchitectures that are designed to better extract the relevantpathological features, either manually designed or by using a neuralarchitecture search network, such as NASNet.

The tile-based training architecture 1800, with tile selectioncontroller agnostic to neural network configuration, allows us to createa deep learning framework capable of identifying biomarkers for which asingle architecture alone (such as FCN) is less accurate or wouldrequire a more timely process.

Furthermore, because the tile-based training architecture 1800 has afeedback configuration, in some examples, the deep learning framework1802 is able to classify regions in training images, e.g., using an FCNarchitecture, and feed back those classified images as a weaklysupervised training images to the tile selection controller 1822. Forexample, a FCN architecture can be used to first identify specifictissue regions (e.g. tumor, stroma) that can then be used as input intothe weakly supervised training pipeline, such as MIL. The models trainedby a weakly supervised pipeline can then be used in conjunction with theFCN architecture, either to verify or improve the FCN architectureresults, or to supplement the FCN architecture by detecting newfeatures.

Further, there are biomarkers for which annotations are not possible,for example previously undiscovered tissue features that are correlatedwith genotypes, gene expression, or patient metadata. In such cases, thegenotypes, gene expression, or patient metadata can be used to createthe slide-level labels that are used to train a secondary model or FCNarchitecture itself using a weakly supervised framework to detect thenew classification.

Further still, the architecture 1800 is able to provide tissue andtissue artifact detection. The regions in a slide image that containtissue may first be detected to be used as input into the FCNarchitecture model. Imaging techniques such as color or texturethresholding can be used to identify tissue regions, and the deeplearning convolutional neural network models herein (e.g., the FCNarchitecture) can be used to further improve on the generalizability andaccuracy of the tissue detection. Color or texture thresholding can alsobe used to identify spurious artifacts within the tissue image, andweakly supervised deep learning techniques can be used to improvegeneralizability and accuracy.

Yet further still, the architecture 1800 is able to provide markerdetection. Histopathology images may contain notations or annotationsdrawn by pathologists on the slide with a marker, for example toindicate micro/macro-dissection regions where tissue DNA/RNA analysisshould be performed. A marker detection model, similar to a tissuedetection model, can be used to identify which regions have been chosenby pathologists for analysis. This would further supplement the dataprocessing for weakly supervised training to isolate those regions whereDNA/RNA analysis was performed that result in the slide-level labels.

In FIG. 28, a process 2800 is provided for determining a proposedimmunotherapy treatment for a patient using the imaging-based biomarkerpredictor system 102 of FIG. 1, and in particular the biomarkerprediction of the deep learning framework 300 of FIG. 3. Initially,histopathology images such as stained H&E images are received at thesystem 102 (2802). At a process 2804, each histopathology image isapplied to a trained deep learning framework, such as one implementingone or more FCN classification configurations described herein. At aprocess 2806, the trained deep learning framework applies the images toa trained tissue classifier model and a trained biomarker segmentationmodel to determine biomarker status of the tissue regions of the image.In some examples, a trained cell segmentation classifier model isfurther used by the process 2806. The process 2806 generates biomarkerstatus and biomarker metrics for the image. As shown in FIG. 29, theoutput from the process 2806 may be provided to a process 2808 andimplemented on a tumor therapy decision system 2900 (such as may be partof a genomic sequencing system, oncology system, chemotherapy decisionsystem, immunotherapy decision system, or other therapy decision system)that determines a tumor type based on the received data, including basedon the biomarker metrics, genomic sequencing data, etc. The system 2900analyzes the biomarker status and/or biomarker metrics and otherreceived molecular data against available immunotherapies 2902, at aprocess 2810, and the system 2900 recommends a matched listing ofpossible tumor-type specific immunotherapies 2904, filtered from thelist of available immunotherapies 2902, in the form of a matched therapyreport.

In various examples, the imaging-based biomarker prediction systemsherein may be deployed partially or wholly within a dedicated slideimager, such as a high throughput digital scanner. FIG. 30 illustratesan example system 3000 having a dedicated ultrafast pathology (slide)scanner system 3002, such as a Philips IntelliSite Pathology Solutionavailable from Koninklijke Philips N. V. of Amsterdam, Netherlands. Insome examples, the pathology scanner system 3002 may contain a pluralityof trained biomarker classification models. Exemplary models mayinclude, for instance, those disclosed in U.S. application Ser. No.16/412,362. The scanner system 3002 is coupled to an imaging-basedbiomarker prediction system 3004, implementing processes as discussedand illustrated in examples herein. For example, in the illustratedexample, the system 3004 includes a deep learning framework 3006 basedon tile-based multiscale and/or single-scale classification modules, inaccordance with examples herein, having one or more trained biomarkerclassifiers 3008, a trained cell classifier 3010, and a trained tissueclassifier 3012. The deep learning framework 3006 performs biomarker andtumor classifications on histopathology images and stores theclassification data as overlay data with the original images in agenerated images database 3014. The images may be saved as TIFF files,for example. Although the database 3014 may include JSON files and otherdata generated by the classification processes herein. In some examples,the deep learning framework may be integrated in whole or in part withinthe scanner 3002, as shown in optional block 3015.

To manage generated images, which can be quite large, an imagemanagement system and viewer generator 3016 is provided. In theillustrated example, the system 3016 is illustrated as external to theimaging-based biomarker prediction system 3004, connected by a privateor public network. Yet, in other examples, all or part of the system3016 may be deployed in the system 3004, as shown at 3019. In someexamples, the system 3016 is cloud based, and stores generated imagesfrom (or instead of) the database 3014. In some examples, the system3016 generates a web-accessible cloud based viewer, allowingpathologists to access, view, and manipulate, through a graphic userinterface, histopathology images with various classification overlays,examples of which are illustrated in FIGS. 31-37.

In some examples, the image management system 3016 manages receipt ofscanned slide images 3018 from the scanner 3002, where these slideimages are generated from an imager 3020.

In the illustrated example, the image management system 3016 generatesan executable viewer App 3024 and deploys that App 3024 to an AppDeployment Engine 3022 of the scanner 3002. The App Deployment Engine3022 may provide functionality such as GUI generation allowing users tointeract with the view App 3024, an App marketplace allowing users todownload the viewer App 3024 from the image management system 3016 orfrom other network accessible sources.

FIGS. 31-37 illustrate various digital screenshots generated by theembedded viewer 3024, in an example, where these screenshots arepresented in a GUI format that allows users to interact with thedisplayed images for zooming in and out and for displaying differentclassifications of tissue, cells, biomarker, and/or tumors.

Referring to FIG. 31, a GUI generated display 3100 having a panel 3102showing an entire histopathology image 3104 and an enlarged portion(1.3× zoom factor) of that image 3104 displayed as window 3106. Thepanel 3102 further includes a magnification factor corresponding to thewindow 3106 and a tumor content report. FIG. 32 illustrates the display3100, but after a user as zoomed in on the window 3106, to a 3.0× zoomfactor. FIG. 33 is similar but at a 5.7× zoom factor. FIG. 34illustrates a drop down menu 3108 listing a series of classificationsthat a user can select for generating an classification overlay map thatwill be displayed on the display 3100. FIG. 35 illustrates the resultingdisplay 3100 with an overlay map showing tumor classified tissue,demonstrating, in this example, that the tissue had been divided intotiles, and the tiles having classifications are shown. In the example ofFIG. 35, the classification illustrated is a tumor classification. FIG.36 illustrates another example classification overlay mapping, this oneof a cell classification, epithelium, immune, stroma, tumor, or other.FIG. 37 illustrates a magnified cell classification overlay mapping thatshowing classifications may indeed may be displayed at a magnificationsufficient to differential different cells with an histopathology image.

Smart Pathology

The foregoing embodiments may be orchestrated to assist a pathologistduring the course of a pathology review and reporting for one or moreslides from a subject's specimen. Pathologists may receive one or moreslides, such as H&E slides or IHC slides which have been prepared by alaboratory technician from a tissue, such as the cancerous tumor tissuefrom a subject biopsied to identify a diagnosis. The pathologist mayreview the one or more slides and identify a diagnosis of the cancer forthe subject from features of the cells and tissue structures identifiedtherein.

Pathologists may benefit from incorporating an interactive softwareportal having one or more of the artificial intelligence enginesdescribed herein. In one example, a pathologist may log in to a softwareusing a username and password associated with a specific user having aset of privileges. Privileges may include a cohort of subjects for whichthe pathologist may conduct a review, diagnosis, and reporting for, oneor more default parameters such as user preferences, user history, anduser workflow requirements.

User preferences may include, for example, one or more artificialintelligence engines, overlays, colors schemes as set for the user. Inone example, a user may desire that their outlines for a tumor tissuedissection should be green, while another user may prefer yellow, orwhite. In another example, a user may desire to view more than oneoverlay at a time, or view predictions from one or more biomarkers foran overlay. Such a user may select a specific color scheme for eachoverlay to assist in visually separating the information displayed in amanner that is intuitive to the user. It should be understood that tumortissue dissection includes dissection (dissection with the use of amicroscope) and dissection (dissection without the use of a microscope.For the purposes of this disclosure, any reference of dissectionincludes both micro and macro dissection techniques.

A user history may include one or more subjects for which the user hasinitiated, completed, or otherwise acted upon. A user may select thesubject and slide from their history to view the reporting theypreviously completed or to continue reporting where they left offpreviously.

User workflow requirements may be set by an administrator, officescheduler, pathologist, supervisor, project manager, or other authorizedusers to identify one or more actions to be taken by the user. In oneexample a workflow may include a preprocessing order to, for example,slice 10 slides of tissue from a FFPE tissue block. A workflow mayfurther include a verification order to identify the amount of viablecells present in the slides and evaluate their efficacy for sequencingthe included cells for DNA or RNA NGS sequencing. A workflow may furtherinclude a dissection order to identify one or more dissections necessaryin the 10 slides to ensure a comparable amount of viable DNA/RNAspecimen for sequencing. A workflow may further comprise an order forevaluating one or more slides for biomarkers. A workflow may concludewith an order for generating a pathology report, scraping the tissuespecimen from the slide, and sending the scraped specimen to a beprepared for next generation sequencing.

A system operating according to the above will empower pathologists witha more informed choice of number slides to scrape and areas to dissectin addition to their own observations. Furthermore, the system willreduce tissue waste by achieving more accurate DNA yields in the firstround of processing. Higher quality first round processing also reducedthe need of re-extraction, which in turn reduces overall turn aroundtime for next-generation sequencing.

Pathologists may interact with the system during review of histologyslides just before they are scraped and DNA/RNA is extracted. In oneexample, the slides may be digitized before review. Digitized slides maybe processed to estimate yield and predict a suggested quantity ofslides to scrape. At the same time, the slide may be processed toidentify one or more dissection areas. The results of estimated yieldand dissection outlines may be visually presented to the pathologist viaa browser-based user interface (UI). In one embodiment, the UI may beintegrated with the viewer, described above with respect to FIGS. 31-37so that the quality, yield, number of slides, and dissection predictionsmay be presented along with the slide image and any other overlays.

Nucleic acid yield may be determined for each slide in order toidentify, for example a target total nucleic acid yield between 50ng-2000 ng. The mass may be determined in advance, selected by thepathologist at run time, stored in the user preferences, or selectedbased upon the order workflow for the specimen and the sequencing thatmay be performed. The pathologists may also decide if the sample shouldbe dissected and indicate the area for dissection by drawing on theslide. The area selected may contain at least a threshold amount oftumor cells, such as 20%, 25%, 30%, or another threshold as desired toensure adequate sampling during sequencing. Thresholds may be based onestimates of nucleic acid yield. In one example, a slide that isestimated to produce less than 50 ng may be discarded and/or a newspecimen requested from the physician. Other yields may also beconsidered as the state of the measuring equipment improves. A targettotal nucleic acid yield may be determined by the summation of DNA andRNA molecules in the cell. For example, a cell may include The thresholdamount of tumor cells may be determined in advance, selected by thepathologist at run time, stored in the user preferences, or selectedbased upon the order workflow for the specimen and the sequencing thatmay be performed. In ordinary processing, these estimates may beperformed entirely by visual inspection. Unsurprisingly, the resultingDNA yields were sometimes too low, because not enough slides werescraped, or too high, because too many slides were scraped. Theadvantages of an artificial intelligence engine assisted review maynegate many of the incorrect yield sizes by allowing the pathologist toconsult for a ‘second opinion’ to their own estimate, or as a first lineof evaluation.

A single tumor cell contains about 6 pg (pico-grams) of DNA and about10-30 pg of RNA, allowing a per cell yield to be estimated for totalnucleic acid (TNA=DNA+RNA) by counting all the tumor cells and immunecells in the dissection area and multiplying by a predetermined averagecell values between 16 and 36 pg. However, because the amount of RNA percell is variable, and tissue can be degraded from age or chemicaltreatment, the cell counts alone may not always be the most accuratepredictor of TNA. By including additional imaging features beyond justcell count, a stronger correlation to the actual TNA may be estimated.Accordingly, using the cell imaging feature extraction disclosed herein,including, for example, cell segmentation imaging features of FIG. 7,above, TNA estimation may be improved. In some embodiments, average cellTNA are adjusted to account for the type of tumor the specimen isidentified as having as different cancerous tumor cells have differentaverage TNAs. Additional imaging features can be broken into threegeneral categories: tumor shape features (area, perimeter, circularity,number of tumor chunks, density, etc. for the entire tumor), cell shapefeatures (area, perimeter, circularity, density, etc. per cell), andcell texture features (R/B/G colors & grayscale texture patterns such asHaralick features and gradient features, per cell). While cell countsaccount for a majority of the TNA estimation, the additional featuresprovide a stronger estimate to reduce excess generation of slides,costly reprocessing of tumor tissue, and/or requesting new specimen froma physician.

In one embodiment, slides may be viewed one at a time through amicroscope having augmented reality superimposed over the image of theslide, before or after staining to identify different characteristics ofthe tissue such as H&E for cellular and structural characteristics orIHC for protein characteristics as disclosed herein. Characteristicsincluding biomarkers, cell outlines, cell shapes, cell types, andproteins present in the cell. In another embodiment, quantitativeestimates of cell counts, tumor shape, and other imaging features may bederived from additional imaging/biomarker algorithms as detailed herein.In another laboratory, all slides may be scanned, and a digital imagegenerated and stored in a database according to the specimen from whichthe slide was taken. A pathologist may view the digital image of theslide on a device with a screen, such as a computer with monitor, mobiledevice with embedded screen, tablet, laptop, television, or otherviewing device. In some instances, the microscope may also broadcast adigital image of the slide to a monitor. Embodiments may includeoverlaying one or more image analysis profiles over the digital image.

In one example overlays may include artifact detection, quality control,scraping identification, cell and tissue predictions, and assisteddiagnosis. In some instances, notes made by other pathologists for theinstant slide may be overlayed with the slide including one or moremarkers identifying the location on the slide the note corresponds to,if applicable.

When tumor tissues are viewed as part of an overall order processingpipeline for histopathic analysis or genetic sequencing pipeline, theartificial intelligence engine may relay predictions, errors, and otherfindings to the appropriate process of the pipeline. For example, aprocess may be instantiated to notify the ordering physician and/orpatient if a low quality sample, low yield sample, or other deficiencyneeds more turnaround time in the laboratory to generate results.

Templates, as a component of a user workflow, may guide a pathologistthrough a series of models. For example, upon loading a slide for thefirst time, the workflow may load a first overlay identifying thenucleic acid yield for the slide, the overlay may further identify tumorquality and quantity in addition to other cell types present in theslide. If the tumor tissue is of sufficient quality and quantity, asubsequent model may be called for the pathologist review and a statusof the tumor QA passed may be overlaid in a section of the slideunencumbered by tissue and/or corresponding stain.

In one embodiment, a microscope or computing device with a monitor maybe provided to a pathologist as well as a hardware speaker for sendingand receiving commands. A pathologist may physically turn a knob orpress a button to advance through models/overlays or recall previousmodels on demand. When coupled with the hardware speaker, such as thedevice of U.S. patent application Ser. No. 16/852,138, filed Apr. 17,2020, and titled “Collaborative Artificial Intelligence Method AndSystem” incorporated herein by reference for all purposes, thepathologist may verbally call out notes on the slide which are recordedand switch between overlays using voice commands and/or gestures.

In one embodiment, an AI-assisted pathology review (Smart Pathology) mayinclude three models: number of slides prediction (NSP), dissectionflagging (MF), and dissection area prediction (MAP). A model for NSP maygenerate an overlay displaying the prediction of the number of slides toscrape. In one embodiment, the number of slides may be estimated bycomputing imaging features, as described above, from an H&E scannedimage, and predicting the DNA yield using a linear model to multiply thenumber of each type of tumor cells by a respective average nucleic acidmultiplier. In another embodiment, the number of slides prediction iscomputed by dividing a user-defined target total nucleic acid yield bythe predicted TNA yield by presuming that the additional slides arelikely to contain similar TNA yields because slices of tumor tissue froma formalin-fixed paraffin embedded (FFPE) tissue are likely similar toeach other in shape, cellular makeup, and tumor content. In one examplethe imaging features may be the total cell count detected by a celldetection model as described above. The imaging features may alsoinclude tumor shape features, as well as other features derived fromshapes, perimeters, densities, and textures of the detected cellsthemselves. In some embodiments, a model for identifying whether a slideshould receive dissection may generate a dissection flag, which willprompt a dissection model to be run. Using the same imaging featuresused for NSP, a logistic regression model may be trained on slides whichhave either had or not had a dissection. The MF model may then recognizeslides which should have dissection performed thereon. In someembodiments, for example, when a dissection model is not computationallyintensive, the MF model may be combined with the MAP model. A model maygenerate an overlay identifying and drawing a section of tumor tissue tobe dissected from the rest of the tissue on a slide. For example, anoverlay may include, for example, the algorithms for generating the cellclassification overlays of FIGS. 10a-b, 15a-b, 16a-f , 23, 25, 35, or36. Once the tumor tissue is identified, a dissection boundary may bedrawn around it in such a manner that allows for easiermicro/macro-dissection by a person or machine. This may include dilatingthe boundary by a number of pixels to smooth the resulting boundary. Thenumber of pixels may vary depending on the resolution and magnificationof the digital image but may be up to 20 pixels for magnificationshaving a few pixels per cell. In another example, where a model istrained to draw the dissection lines, training a MAP model may includeproviding the digital image of a slide having pathologist annotateddissection lines therein. The dissection area drawn on the digital sliderestricts the scraping to an area that contains at least the tumorthreshold number of tumor cells, for example 20%, or 50 nanograms byvolume of nucleic acid. A MAP model may be provided with a tumor mask asinput and, optionally, the image may be “cleaned” with image processingtechniques that remove small specs (less than 1/10 of the largest tissuesize) and fill holes. The bounds drawn on the slide may then be dilatedto the preprocessed tumor mask by a fixed number of pixels to produce anarea that fully encloses a majority of tumor cells.

FIG. 39 illustrates a flowchart for a Smart Path processing pipeline3900 for automatically processing and generating NSP, MF and MAPpredictions and overlays. The Smart Path processes described in thevarious examples herein may be performed partially or wholly within anentity, such as any of the entities described and/or illustrated inreference to example prediction systems such as in FIGS. 1, 3, 30, 38,and/or 40, and including by way of example a laboratory, a health careinstitution, and/or pharmaceutical company receiving the H&E slides. Atstep 3910, a histology slide is scanned and the Smart Path processingpipeline is initiated. Scanning a histology slide may include scanning anumber of slides in a slide book, the steps herein are repeated for eachslide. At step 3920, the pipeline draws and dilates tumor area mask fordissection. Drawing of tumor area mask may be performed by a trainedmachine learning model, such as a model trained on images of histologyslides having a mask drawn around the tumor cells of the slide. Dilationto the closest cells for the tumor may be performed to reduce theinclusion of additional, non-tumor cells in the dissected area while atthe same time ensuring the shape and size of the resulting dissectionare convenient for a machine or user to scrape. Dilation may be trained,or may be a post-processing step which identifies the outermost edge oftumor cells within the mask area and dilates the overlay mask to shrinkwrap the tumor outer circumference or smooth the tumor area based on theshape. In another example, the dilation may include multiple tumorcentroids (grouping of cells with distinct outer edges). The multipletumor centroids are each evaluated and if they are not smaller than 20%of the area of cells on the slide, may all be included in the dissectionboundary. When multiple centroids are included in the dissectionboundary, dilation of the dissection boundary may include manyadditional non-tumor cells and smoothing may further include additionalcells in order to promote ease of scraping by a user or machine. At step3930, the pipeline extracts slide features and stores a masked subsetand an unmasked subset of them in a structured format. A masked subsetare the features that correspond to the tumor tissue within the tumorarea mask calculated at step 3920 and the unmasked subset are thefeatures that correspond to the whole slide without consideration of thetumor area mask. The masked subset enables the tumor content andestimated nucleic yield to be accurately predicted from within thedissection boundary and the unmasked features enables the comparison ofavailable tumor tissue within the boundary to all tissue on the slide.At step 3940, the pipeline may set the dissection flag if tumor percentis below a threshold. Tumor percentage threshold may be set at 20% toprovide optimal nucleic acid yield or may be set at an estimated nucleicacid yield of 50 ng. In some embodiments, step 3940 may be excluded fromthe processing pipeline, for example, when pathologists prefer to alwayssee the predicted dissection area mask. A user preference may be set toalways show MAP or to only show MAP when the tumor percentage is below adesired threshold. When a pathologist disagrees with the predicteddissection mask overlayed, they may draw their own on the original slideand rescan the slide, or draw the mask on the digital image and save itas an adjusted mask. At step 3950, the pipeline calculates the number ofslides necessary to scrape for quality control (QC) and sequencingpasses. The number of slides necessary may be customized by selectingdiffering total target yields in nanograms (ng). For example apathologist may need to conserve tissue so selects a yield of 50 ng,select a standard yield of 250 ng, or elect to increase the yield to 400ng ensure enough nucleic acid is collected to ensure a successfulsequencing. It should be understood that target yields may vary within arange of 50-2000 ng and as technology improves, may reduce below 50 ng.The number of slides may be generated by dividing the target totalnucleic yield selected by the predicted yield of the first slide. Atstep 3960, the pipeline stores each generated prediction and overlay forthe scanned slide. A pathologist may retrieve these overlays whenaccessing the digital image for review at their workstation.

A system, such as described herein, may further track performancemetrics to evaluate the success of the system in reducing the number ofreruns, overall processing time, and improving the accuracy of thepathologist review. Performance metrics may include collection date,review user, review date, total number of slides scraped, the tumorcontent (ng) the pathologist targeted (2000, 1000, 450, 250, 100, 50,etc.), quality control pass/fail, tumor purity, pathologist overrulemodel prediction, number of re-extraction attempts, NGS sequencingsuccess/fail, turn around time for sequencing order, and record ofpathologist drawn dissection lines as compared to model predicted whenpathologist draws their own. Metrics may be referenced when retrainingany models based upon whether the pathologists used the overlayedpredictions, replaced the predicted dissection with a new, hand-drawnone, or if the processing time was reduced due to fewer re-sequenceswhen sequencing or a quality control metric fails.

FIG. 40 is an illustration of a Smart Pathology pipeline for generatingdissection boundaries, estimating dissection yields, and predicting anumber of slides, in accordance with an example.

A pathologist may prepare a slide 4010 by staining it with H&E inpreparation for their pathology review and diagnosis. As part of thepreparation, the stained slide may be scanned and uploaded to a SmartPathology pipeline 4020 having a dissector module 4022, total nucleicacid yield predictor module 4024, and a number of slides predictormodule 4026. A dissector module 4022 may apply a cell segmentation, suchas the cell segmentation of FIG. 7 above, to identify the cell typeswithin the slide. The dissector module 4022 may further draw adissection boundary around the largest grouping of tumor cells,excluding any cells which are too small to be included. A determinationof whether tumor cells are too small to be included may be based off ofthe number of cells in the largest grouping of tumor cells as comparedto the additional number of tumor cells and/or additional number ofnon-tumor cells to be included by adding each additional grouping oftumor cells. When the largest grouping of tumor cells satisfy a 50 ngminimum threshold, and the next largest grouping of tumor cells do notadd 5 ng or more, they may be excluded and the dissection boundary drawnonly around the largest grouping of cells. In another embodiment, eachindividual grouping of cells may be estimated for TNA and the largestgrouping with close proximity to one another may be selected to maximizethe tumor cells while minimizing the non-tumor cells. Close proximitymay be defined by width of the cutting and scraping blade or width ofthe slide. For example, if a dissection area is less than 10% the widthof the slide, or less than the width of the scraping blade, then thedissection area may not be included. In an example where the non-tumorcells substantially outnumber the tumor cells for a grouping of tumorcells to be included, that grouping of tumor cells may not be added tothe dissection boundary. Once the dissection boundary has beengenerated, a total nucleic acid yield predictor module 4024 may identifythe TNA for the tumor cells within the boundary. Identifying the TNA mayinclude summing the total nucleic yields of the groupings of tumor cellsincluded in the dissection boundary, counting the number of tumor cellspresent and multiplying them by an average TNA factor for each cell(such as a value between 16 and 30 pg), calculating the surface area oftumor cells and multiplying the area by a unit area TNA average yield,or applying an artificial intelligence engine trained to predict theTNA. In one example, the trained model may include slides which havebeen scraped and tested for TNA, the cancer diagnosis identifying thetype of tumor cell, the size, perimeter, shape, and color of eachindividual cell, and the other imaging features from the analysis of thedigital image of the slide to predict the TNA from the type, quantity,and density of the tumor cells. Once the TNA for slide 4010 has beendetermined, the number of slides predictor module 4026 may generate apredicted number of slides that should be prepared and scraped forsequencing which will satisfy a target total nucleic yield. In oneexample, a target TNA yield may be selected between 50 and 2000 ng basedon the precision of the TNA measuring device/method and thepathologist's interests in ensuring more than enough nucleic acid ispresent for sequencing as weighed against the on-hand FFPE tumor. Themore available tumor to sequence, the more likely a pathologist is toinclude a higher threshold for target TNA yield. The number of slidesmay then be predicted by dividing the selected target TNA yield by theestimated TNA yield of the tumor cells within the dissection boundaryrounded up to the nearest integer. Slide 4030 illustrates one example ofan overlay to display the information to the pathologist. Slide 4030includes a visual representation of the dissection boundary (asillustrated by a black outline), the selected TNA yield (400 ng), thepredicted TNA yield of the tumor cells within the dissection boundary(112 ng), and the number of predicted slides (400/112=3.6, rounded up to4). The predicted number of slides and the dissection outline may beprovided to slide preparation 4040, wherein an automated slidepreparation device 4042 may automatically retrieve the tumor FFPE blockfrom cold storage, slice the number of slides predicted, and prepareslides 4050 a-d for sequencing; or a lab technician, the pathologist, orother qualified individual may receive a notification as part of a userworkflow as described above from a manual slide preparation 4044, toprepare the number of predicted slides for sequencing. In someinstances, the original, stained slide 4010 may be included in the totalnumber of slides, for example, when the laboratory sequences bothstained and unstained slides, and slide 4050 d may not be prepared atslide preparation 4040. As used herein, unstained slides refers toslides sequentially cut from the same formalin-fixed paraffin-embeddedtissue block.

FIG. 41 is an illustration of a Smart Pathology generated graphical userinterface (GUI) display as may be generated by systems such as thesystems of FIGS. 1, 3, 30, 38, and 40.

The GUI 4100 may display to a pathologist images of the scanned slidewith outlines indicating the whole slide scraping content as well as thedissection boundary that may be scraped when the accuracy of thesubsequent sequencing may be improved by having a higher tumor contentpercentage of the cells. GUI 4100 may display, for example, the wholeslide boundary as a black outline and the dissection boundary as a greenoutline. The colors, thickness of the line, opacity and othercharacteristics may be set in the user preferences. GUI 4100 may alsoinclude one or more estimates for the predicted TNA yield and number ofslides to be prepared and extracted based on whether the whole slide isscraped or the slide is scraped within the dissection boundary only. Asillustrated, target TNAs of 100, 400, and 1000 ng may be presented, or adropdown (not displayed) or text box (not displayed) may be used tospecifically identify a single target TNA from which to calculate thewhole slide predicted TNA yield or the dissection boundary predicted TNAyield.

FIG. 38 illustrates an example computing device 3800 for implementingthe imaging-based biomarker prediction system 100 of FIG. 1. Asillustrated, the system 100 may be implemented on the computing device3800 and in particular on one or more processing units 3810, which mayrepresent Central Processing Units (CPUs), and/or on one or more orGraphical Processing Units (GPUs) 3811, including clusters of CPUsand/or GPUs, and/or one or more tensor processing unites (TPU) (alsolabeled 3811), any of which may be cloud based. Features and functionsdescribed for the system 100 may be stored on and implemented from oneor more non-transitory computer-readable media 3812 of the computingdevice 3800. The computer-readable media 3812 may include, for example,an operating system 3814 and the deep learning framework 3816 havingelements corresponding to that of deep learning framework 300, includingthe pre-processing controller 302, classifier modules 304 and 306, andthe post-processing controller 308. More generally, thecomputer-readable media 3812 may store trained deep learning models,executable code, etc. used for implementing the techniques herein. Thecomputer-readable media 3812 and the processing units 3810 andTPU(S)/GPU(S) 3811 may store image data, tissue classification data,cell segmentation data, lymphocyte segmentation data, TILs metrics, andother data herein in one or more databases 3813. The computing device3800 includes a network interface 3824 communicatively coupled to thenetwork 3850, for communicating to and/or from a portable personalcomputer, smart phone, electronic document, tablet, and/or desktoppersonal computer, or other computing devices. The computing devicefurther includes an I/O interface 3826 connected to devices, such asdigital displays 3828, user input devices 3830, etc. In some examples,as described herein, the computing device 3800 generates biomarkerprediction as an electronic document 3815 that can be accessed and/orshared on the network 3850. In the illustrated example, the system 100is implemented on a single server 3800. However, the functions of thesystem 100 may be implemented across distributed devices 3800, 3802,3804, etc. connected to one another through a communication link. Inother examples, functionality of the system 100 may be distributedacross any number of devices, including the portable personal computer,smart phone, electronic document, tablet, and desktop personal computerdevices shown. In other examples, the functions of the system 100 may becloud based, such as, for example one or more connected cloud TPU (s)customized to perform machine learning processes. The network 3850 maybe a public network such as the Internet, private network such asresearch institution's or corporation's private network, or anycombination thereof. Networks can include, local area network (LAN),wide area network (WAN), cellular, satellite, or other networkinfrastructure, whether wireless or wired. The network can utilizecommunications protocols, including packet-based and/or datagram-basedprotocols such as internet protocol (IP), transmission control protocol(TCP), user datagram protocol (UDP), or other types of protocols.Moreover, the network can include a number of devices that facilitatenetwork communications and/or form a hardware basis for the networks,such as switches, routers, gateways, access points (such as a wirelessaccess point as shown), firewalls, base stations, repeaters, backbonedevices, etc.

The computer-readable media may include executable computer-readablecode stored thereon for programming a computer (e.g., comprising aprocessor(s) and GPU(s)) to the techniques herein. Examples of suchcomputer-readable storage media include a hard disk, a CD-ROM, digitalversatile disks (DVDs), an optical storage device, a magnetic storagedevice, a ROM (Read Only Memory), a PROM (Programmable Read OnlyMemory), an EPROM (Erasable Programmable Read Only Memory), an EEPROM(Electrically Erasable Programmable Read Only Memory) and a Flashmemory. More generally, the processing units of the computing device1300 may represent a CPU-type processing unit, a GPU-type processingunit, a TPU-type processing unit, a field-programmable gate array(FPGA), another class of digital signal processor (DSP), or otherhardware logic components that can be driven by a CPU.

It is noted that while example deep learning frameworks herein have beendescribed as configured with example machine learning architectures (FCNconfigurations), any number of suitable convolutional neural networkarchitectures may be used. Broadly speaking, the deep learningframeworks herein may implement any suitable statistical model (e.g., aneural network or other model implemented through a machine learningprocess) that will be applied to each of the received images. Asdiscussed herein, that statistical model may be implemented in a varietyof manners. In some examples, machine learning is used to evaluatetraining images and develop classifiers that correlate predeterminedimage features to specific categories of TILs status. In some examples,image features can be identified as training classifiers using alearning algorithm such as Neural Network, Support Vector Machine (SVM)or other machine learning process. Once classifiers within thestatistical model are adequately trained with a series of trainingimages, the statistical model may be employed in real time to analyzesubsequent images provided as input to the statistical model forpredicting biomarker status. In some examples, when a statistical modelis implemented using a neural network, the neural network may beconfigured in a variety of ways. In some examples, the neural networkmay be a deep neural network and/or a convolutional neural network. Insome examples, the neural network can be a distributed and scalableneural network. The neural network may be customized in a variety ofmanners, including providing a specific top layer such as but notlimited to a logistics regression top layer. A convolutional neuralnetwork can be considered as a neural network that contains sets ofnodes with tied parameters. A deep convolutional neural network can beconsidered as having a stacked structure with a plurality of layers. Theneural network or other machine learning processes may include manydifferent sizes, numbers of layers and levels of connectedness. Somelayers can correspond to stacked convolutional layers (optionallyfollowed by contrast normalization and max-pooling) followed by one ormore fully-connected layers. For neural networks trained by largedatasets, the number of layers and layer size can be increased by usingdropout to address the potential problem of overfitting. In someinstances, a neural network can be designed to forego the use of fullyconnected upper layers at the top of the network. By forcing the networkto go through dimensionality reduction in middle layers, a neuralnetwork model can be designed that is quite deep, while dramaticallyreducing the number of learned parameters.

A system for performing the methods described herein may include acomputing device, and more particularly may be implemented on one ormore processing units, for example, Central Processing Units (CPUs),and/or on one or more or Graphical Processing Units (GPUs), includingclusters of CPUs and/or GPUs. Features and functions described may bestored on and implemented from one or more non-transitorycomputer-readable media of the computing device. The computer-readablemedia may include, for example, an operating system and softwaremodules, or “engines,” that implement the methods described herein. Moregenerally, the computer-readable media may store batch normalizationprocess instructions for the engines for implementing the techniquesherein. The computing device may be a distributed computing system, suchas an Amazon Web Services cloud computing solution.

The computing device includes a network interface communicativelycoupled to network, for communicating to and/or from a portable personalcomputer, smart phone, electronic document, tablet, and/or desktoppersonal computer, or other computing devices. The computing devicefurther includes an I/O interface connected to devices, such as digitaldisplays, user input devices, etc.

The functions of the engines may be implemented across distributedcomputing devices, etc. connected to one another through a communicationlink. In other examples, functionality of the system may be distributedacross any number of devices, including the portable personal computer,smart phone, electronic document, tablet, and desktop personal computerdevices shown. The computing device may be communicatively coupled tothe network and another network. The networks may be public networkssuch as the Internet, a private network such as that of a researchinstitution or a corporation, or any combination thereof. Networks caninclude, local area network (LAN), wide area network (WAN), cellular,satellite, or other network infrastructure, whether wireless or wired.The networks can utilize communications protocols, includingpacket-based and/or datagram-based protocols such as Internet protocol(IP), transmission control protocol (TCP), user datagram protocol (UDP),or other types of protocols. Moreover, the networks can include a numberof devices that facilitate network communications and/or form a hardwarebasis for the networks, such as switches, routers, gateways, accesspoints (such as a wireless access point as shown), firewalls, basestations, repeaters, backbone devices, etc.

ADDITIONAL ASPECTS

Aspect 1: A computer-implemented method for qualifying a specimenprepared on one or more hematoxylin and eosin (H&E) slides and forproviding associated unstained slides for nucleic acid analysis, themethod comprising:

generating a digital image of each of the one or more H&E slidesprepared from the specimen at an image-based nucleic acid yieldprediction system having one or more processors; and

for each digital image:

-   -   identifying, using the one or more processors, tumor cells of        the H&E slide from the digital image;    -   generating a tumor area mask based at least in part on the        identified tumor cells, wherein the tumor area mask defines a        dissection boundary of the H&E slide associated with the digital        image;    -   predicting an expected yield of nucleic acid for the tumor cells        within the dissection boundary; and    -   providing one or more of the associated unstained slides for        nucleic acid analysis when the summation of expected yield of        nucleic acid predicted for each digital image exceeds a        predetermined threshold.

Aspect 2: The method of Aspect 1, wherein only associated unstainedslides with the H&E slides having predicted expected yield of nucleicacid exceeding a slide yield threshold are included in the summation.

Aspect 3: The method of Aspect 1, wherein providing the associatedunstained slides for nucleic acid analysis comprises sending theassociated unstained slides to a third party for sequencing.

Aspect 4: The method of Aspect 1, wherein providing the associatedunstained slides for nucleic acid analysis comprises sequencing thenucleic acids extracted from within the dissection boundary identifiedfrom the H&E slides.

Aspect 5: The method of Aspect 1, wherein the predetermined threshold isbased at least in part on one or more specification requirements.

Aspect 6: The method of Aspect 5, wherein specification requirements arebased on a minimum surface area within the dissection boundary.

Aspect 7: The method of Aspect 5, wherein specification requirements arebased on a minimum cell count within the dissection boundary.

Aspect 8: The method of Aspect 5, wherein specification requirements arebased on a minimum cell density within the dissection boundary.

Aspect 9: The method of Aspect 5, wherein specification requirements areset by an entity.

Aspect 10: The method of Aspect 1, wherein the predetermined thresholdis within a range between and including 50 ng-2000 ng.

Aspect 11: The method of Aspect 1, wherein predicting the expected yieldof nucleic acid for the tumor cells within the dissection boundaryfurther comprises:

providing the digital image of the H&E slide to a model trained on aplurality of H&E slides having labeled dissection boundaries and labeledtotal nucleic yield.

Aspect 12: The method of Aspect 1, wherein predicting the expected yieldof nucleic acid for the tumor cells within the dissection boundaryfurther comprises:

counting the number of tumor cells identified within the dissectionboundary and multiplying the count by a tumor cell average nucleic acidyield.

Aspect 13: The method of Aspect 1, wherein predicting the expected yieldof nucleic acid for the tumor cells within the dissection boundaryfurther comprises:

calculating the surface area of the dissection boundary and multiplyingthe surface area by a dissection boundary average nucleic acid yield.

Aspect 14: The method of Aspect 1, wherein the dissection boundary is amicrodissection boundary.

Aspect 15: The method of Aspect 1, wherein the dissection boundary is amacrodissection boundary.

Aspect 16: The method of Aspect 1, wherein the dissection boundary is awhole slide boundary.

Aspect 17: The method of Aspect 1, wherein identifying the tumor cellsof the H&E slide from the digital image comprises identifying the tumorcells using a trained cell segmentation model.

Aspect 18: The method of Aspect 17, wherein identifying the tumor usingthe trained cell segmentation model comprises:

applying, using the one or more processors, a plurality of tile imagesformed from the digital image to the trained cell segmentation modeland, for each tile, assigning a cell classification to one or morepixels within the tile image.

Aspect 19: The method of Aspect 18, wherein assigning the cellclassification to one or more pixels within the tile image comprises:

identifying, using the one or more processors, the one or more pixels asa cell interior, a cell border, or a cell exterior and classifying theone or more pixels as the cell interior, the cell border, or the cellexterior.

Aspect 20: The method of Aspect 17, wherein the trained cellsegmentation model is a pixel-resolution three-dimensional UNetclassification model trained to classify a cell interior, a cell border,and a cell exterior.

Aspect 21: The method of Aspect 17, wherein identifying the tumor cellsusing the trained cell segmentation model comprises:

applying, using the one or more processors, each of a plurality of tileimages formed from the digital image to the trained cell segmentationmodel; and

performing, using the one or more processors, a registration onsegmented cells in each of the tile images by determining a cell borderof each cell, determining a centroid of each cell, and shiftingcoordinates of centroids to a universal coordinate space for the digitalimage.

Aspect 22: The method of Aspect 17, wherein the trained cellsegmentation model is trained using a set of H&E slide training imagesannotated with identified cell borders, identified cell interiors, andidentified cell exteriors.

Aspect 23: The method of Aspect 1, further comprising superimposing,using a viewer, the tumor area mask over the digital image to visuallyindicate to a user which tumor cells to scrape.

Aspect 24: The method of Aspect 1, wherein generating the tumor areamask further comprises providing the digital image of the H&E slide to amodel trained on a plurality of H&E slides having dissection labels.

Aspect 25: A system comprising a pathology slide scanner systemconfigured to perform the method of Aspect 1.

Aspect 26: The system of Aspect 25, wherein the pathology slide scannersystem is communicatively coupled to an image-based tumor cellsprediction system through a communication network for providing the oneor more of the associated unstained slides for nucleic acid analysiswhen the summation of expected yield of nucleic acid predicted for eachdigital image exceeds the predetermined threshold.

Aspect 27: The method of Aspect 1, wherein the one or more processorsare one or more graphics processing units (GPUs), tensor processingunits (TPUs), and/or central processing units (CPUs).

Aspect 28: The method of Aspect 1, wherein the expected yield of nucleicacid is confirmed through sequencing the tumor cells within thedissection boundary.

Aspect 29: The method of Aspect 28, where sequencing is next-generationsequencing.

Aspect 30: The method of Aspect 28, where sequencing is short-readsequencing.

Aspect 31: A computer-implemented method comprising: receiving a firsthematoxylin and eosin (H&E) slide prepared from a tumor block at animage-based nucleic acid yield prediction system having one or moreprocessors; identifying, using the one or more processors, tumor cellsof the H&E slide from a digital image; generating a tumor area maskbased at least in part on the identified tumor cells, wherein the tumorarea mask defines a dissection boundary of the H&E slide associated withthe digital image; predicting an expected yield of nucleic acid for thetumor cells within the dissection boundary; based on the predictedexpected yield of nucleic acid, identifying a number of H&E slides toprepare from the tumor block to satisfy a total nucleic yield; andpreparing the number of H&E slides from the tumor block when thepredicted expected yield of nucleic acid exceeds a predeterminedthreshold.

Aspect 32: The method of Aspect 1, wherein the predetermined thresholdis based at least in part on one or more specification requirements.

Aspect 33: The method of Aspect 32, wherein specification requirementsare based on a minimum surface area within the dissection boundary.

Aspect 34: The method of Aspect 32, wherein specification requirementsare based on a minimum cell count within the dissection boundary.

Aspect 35: The method of Aspect 32, wherein specification requirementsare based on a minimum cell density within the dissection boundary.

Aspect 36: The method of Aspect 32, wherein specification requirementsare set by an entity.

Aspect 37: The method of Aspect 31, wherein the predetermined thresholdis selected from a range between and including 50 ng-2000 ng.

Aspect 38: The method of Aspect 31, wherein predicting the expectedyield of nucleic acid for the tumor cells within the dissection boundarycomprises: providing the digital image of the H&E slide to a modeltrained on a plurality of H&E slides having labeled dissectionboundaries and labeled total nucleic yield.

Aspect 39: The method of Aspect 31, wherein predicting the expectedyield of nucleic acid for the tumor cells within the dissection boundarycomprises: counting the number of tumor cells identified within thedissection boundary and multiplying the count by a tumor cell averagenucleic acid yield.

Aspect 40: The method of Aspect 31, wherein predicting the expectedyield of nucleic acid for the tumor cells within the dissection boundarycomprises: calculating the surface area of the dissection boundary andmultiplying the surface area by a dissection boundary average nucleicacid yield.

Aspect 41: The method of Aspect 31, wherein the dissection boundary is amicrodissection boundary.

Aspect 42: The method of Aspect 31, wherein the dissection boundary is amacrodissection boundary.

Aspect 43: The method of Aspect 31, wherein the dissection boundary is awhole slide boundary.

Aspect 44: The method of Aspect 31, wherein identifying the tumor cellsof the H&E slide from the digital image comprises identifying the tumorcells using a trained cell segmentation model.

Aspect 45: The method of Aspect 44, wherein identifying the tumor usingthe trained cell segmentation model comprises: applying, using the oneor more processors, a plurality of tile images formed from the digitalimage to the trained cell segmentation model and, for each tile,assigning a cell classification to one or more pixels within the tileimage.

Aspect 46: The method of Aspect 45, wherein assigning the cellclassification to one or more pixels within the tile image comprises:identifying, using the one or more processors, the one or more pixels asa cell interior, a cell border, or a cell exterior and classifying theone or more pixels as the cell interior, the cell border, or the cellexterior.

Aspect 47: The method of Aspect 44, wherein the trained cellsegmentation model is a pixel-resolution three-dimensional UNetclassification model trained to classify a cell interior, a cell border,and a cell exterior.

Aspect 48: The method of Aspect 44, wherein identifying the tumor cellsusing the trained cell segmentation model comprises: applying, using theone or more processors, each of a plurality of tile images formed fromthe digital image to the trained cell segmentation model; andperforming, using the one or more processors, a registration onsegmented cells in each of the tile images by determining a cell borderof each cell, determining a centroid of each cell, and shiftingcoordinates of centroids to a universal coordinate space for the digitalimage.

Aspect 49: The method of Aspect 44, wherein the trained cellsegmentation model is trained using a set of H&E slide training imagesannotated with identified cell borders, identified cell interiors, andidentified cell exteriors.

Aspect 50: The method of Aspect 31, further comprising superimposing,using a viewer, the tumor area mask over the digital image to visuallyindicate to a user which tumor cells to scrape.

Aspect 51: The method of Aspect 31, wherein generating the tumor areamask further comprises providing the digital image of the H&E slide to amodel trained on a plurality of H&E slides having dissection labels.

Aspect 52: A system comprising a pathology slide scanner systemconfigured to perform the method of Aspect 31.

Aspect 53: The system of Aspect 52, further configured to generate adigital image of each of the one or more H&E slides from the tumorblock.

Aspect 54: The system of Aspect 53, wherein the pathology slide scannersystem is communicatively coupled to an image-based tumor cellsprediction system through a communication network for providing the oneor more of digital images.

Aspect 55: The method of Aspect 31, wherein the one or more processorsare one or more graphics processing units (GPUs), tensor processingunits (TPUs), and/or central processing units (CPUs).

Aspect 56: The method of Aspect 31, wherein the expected yield ofnucleic acid is confirmed through sequencing the tumor cells within thedissection boundary.

Aspect 57: The method of Aspect 56, where sequencing is next-generationsequencing.

Aspect 58: The method of Aspect 56, where sequencing is short-readsequencing.

Aspect 59: A computer-implemented method for predicting an expectedyield of nucleic acid from tumor cells within a dissection boundary on ahematoxylin and eosin (H&E) slide, the method comprising: receiving adigital image of the H&E slide at an image-based nucleic acid yieldprediction system having one or more processors; identifying, using theone or more processors, tumor cells of the H&E slide from digital imageusing a trained cell segmentation model; generating a tumor area maskbased at least in part on the identified tumor cells, wherein the tumorarea mask defines the dissection boundary of the H&E slide; predictingthe expected yield of nucleic acid for the tumor cells within thedissection boundary.

Aspect 60: The method of Aspect 59, further comprising: accepting anassociated unstained slide of the H&E slide for next-generationsequencing when the predicted expected yield of nucleic acid exceeds aminimum threshold.

Aspect 61: The method of Aspect 60, wherein the minimum threshold is 50ng.

Aspect 62: The method of Aspect 1, wherein when the predicted expectedyield of nucleic acid fails to satisfy a target total nucleic acidyield: identifying a number of associated unstained slides thatsatisfies the target total nucleic acid yield; and accepting the numberof associated unstained slides for next-generation sequencing.

Aspect 63: The method of Aspect 62, wherein the target total nucleicyield is selected from a range between and including 50 ng-2000 ng.

Aspect 64: The method of Aspect 62, wherein the associated unstainedslides are flagged for scrapping.

Aspect 65: The method of Aspect 62, wherein the associated unstainedslides comprise tissue from the same formalin-fixed paraffin embeddedspecimen.

Aspect 66: The method of Aspect 62, herein the number of associatedunstained slides is estimated by dividing the target total nucleic yieldby the predicted expected yield of the H&E slide and rounding up to thenearest integer.

Aspect 67: The method of Aspect 66, wherein sequencing is performedusing the scrapings from the associated unstained slides.

Aspect 68: The method of Aspect 62, further comprising superimposing,using a viewer, the tumor area mask over the digital image of theassociated unstained slides to visually indicate to a user which tumorcells to scrape.

Aspect 69: The method of Aspect 62, wherein generating the tumor areamask further comprises providing the digital image of the H&E slide to amodel trained on a plurality of H&E slides having dissection labels.

Aspect 70: The method of Aspect 62, wherein predicting the expectedyield of nucleic acid for the tumor cells within the dissection boundaryfurther comprises: providing the digital image of the H&E slide to amodel trained on a plurality of H&E slides having dissection labels andtotal nucleic yield labels.

Aspect 71: The method of Aspect 59, wherein a viewer superimposes thetumor area mask over the digital image of the associated unstainedslides to visually indicate to a user which tumor cells to scrape.

Aspect 72: The method of Aspect 59, wherein generating the tumor areamask further comprises: providing the digital image of the H&E slide toa model trained on a plurality of H&E slides having dissection labels.

Aspect 73: The method of Aspect 59, wherein predicting the expectedyield of nucleic acid for the tumor cells within the dissection boundaryfurther comprises: providing the digital image of the H&E slide to amodel trained on a plurality of H&E slides having dissection labels andtotal nucleic yield labels.

Aspect 74: The method of Aspect 59, wherein predicting the expectedyield of nucleic acid for the tumor cells within the dissection boundaryfurther comprises: counting the number of tumor cells identified withinthe dissection boundary and multiplying the count by a tumor cellaverage nucleic acid yield.

Aspect 75: The method of Aspect 59, wherein predicting the expectedyield of nucleic acid for the tumor cells within the dissection boundaryfurther comprises: calculating the surface area of the dissectionboundary and multiplying the surface area by a dissection boundaryaverage nucleic acid yield.

Aspect 76: The method of Aspect 59, wherein the dissection boundary is amicrodissection boundary.

Aspect 77: The method of Aspect 59, wherein the dissection boundary is amacrodissection boundary.

Aspect 78: The method of Aspect 59, wherein the dissection boundary is awhole slide boundary.

Aspect 79: The method of Aspect 59, wherein identifying tumor cells ofthe H&E slide from the digital image using the trained cell segmentationmodel comprises: applying, using the one or more processors, a pluralityof tile images formed from the digital image to the trained cellsegmentation model and, for each tile, assigning a cell classificationto one or more pixels within the tile image.

Aspect 80: The method of Aspect 79, wherein assigning the cellclassification to one or more pixels within the tile image comprises:identifying, using the one or more processors, the one or more pixels asa cell interior, a cell border, or a cell exterior and classifying theone or more pixels as the cell interior, the cell border, or the cellexterior.

Aspect 81: The method of Aspect 59, wherein the trained cellsegmentation model is a pixel-resolution three-dimensional UNetclassification model trained to classify a cell interior, a cell border,and a cell exterior.

Aspect 82: The method of Aspect 59, wherein identifying tumor cells ofthe H&E slide from the digital image using the trained cell segmentationmodel comprises: applying, using the one or more processors, each of aplurality of tile images formed from the digital image to the trainedcell segmentation model; and performing, using the one or moreprocessors, a registration on segmented cells in each of the tile imagesby determining a cell border of each cell, determining a centroid ofeach cell, and shifting coordinates of centroids to a universalcoordinate space for the digital image.

Aspect 83: The method of Aspect 59, wherein the trained cellsegmentation model is trained using a set of H&E slide training imagesannotated with identified cell borders, identified cell interiors, andidentified cell exteriors.

Aspect 84: An image-based tumor cells prediction system configured toperform the method of Aspect 59, three image-based tumor cellsprediction system being contained within a pathology slide scannersystem.

Aspect 85: An image-based tumor cells prediction system configured toperform the method of Aspect 59, the image-based tumor cells predictionsystem being contained partially within a pathology slide scanner systemand partially within an external prediction computing systemcommunicatively coupled to the pathology slide scanner system through acommunication network.

Aspect 86: The method of Aspect 59, wherein the one or more processorsare one or more graphics processing units (GPUs), tensor processingunits (TPUs), and/or central processing units (CPUs).

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components or multiple components.

Additionally, certain embodiments are described herein as includinglogic or a number of routines, subroutines, applications, orinstructions. These may constitute either software (e.g., code embodiedon a machine-readable medium or in a transmission signal) or hardware.In hardware, the routines, etc., are tangible units capable ofperforming certain operations and may be configured or arranged in acertain manner. In example embodiments, one or more computer systems(e.g., a standalone, client or server computer system) or one or morehardware modules of a computer system (e.g., a processor or a group ofprocessors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In various embodiments, a hardware module may be implementedmechanically or electronically. For example, a hardware module maycomprise dedicated circuitry or logic that is permanently configured(e.g., as a special-purpose processor, such as a microcontroller, fieldprogrammable gate array (FPGA) or an application-specific integratedcircuit (ASIC)) to perform certain operations. A hardware module mayalso comprise programmable logic or circuitry (e.g., as encompassedwithin a general-purpose processor or other programmable processor) thatis temporarily configured by software to perform certain operations. Itwill be appreciated that the decision to implement a hardware modulemechanically, in dedicated and permanently configured circuitry, or intemporarily configured circuitry (e.g., configured by software) may bedriven by cost and time considerations.

Accordingly, the term “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. Considering embodiments inwhich hardware modules are temporarily configured (e.g., programmed),each of the hardware modules need not be configured or instantiated atany one instance in time. For example, where the hardware modulescomprise a general-purpose processor configured using software, thegeneral-purpose processor may be configured as respective differenthardware modules at different times. Software may accordingly configurea processor, for example, to constitute a particular hardware module atone instance of time and to constitute a different hardware module at adifferent instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multipleof such hardware modules exist contemporaneously, communications may beachieved through signal transmission (e.g., over appropriate circuitsand buses) that connects the hardware modules. In embodiments in whichmultiple hardware modules are configured or instantiated at differenttimes, communications between such hardware modules may be achieved, forexample, through the storage and retrieval of information in memorystructures to which the multiple hardware modules have access. Forexample, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of the example methods described herein can beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions. The modulesreferred to herein may, in some example embodiments, compriseprocessor-implemented modules.

Similarly, the methods or routines described herein may be at leastpartially processor-implemented. For example, at least some of theoperations of a method can be performed by one or more processors orprocessor-implemented hardware modules. The performance of certain ofthe operations may be distributed among the one or more processors, notonly residing within a single machine, but also deployed across a numberof machines. In some example embodiments, the processor or processorsmay be located in a single location (e.g., within a home environment, anoffice environment or as a server farm), while in other embodiments theprocessors may be distributed across a number of locations.

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but also deployed across a number of machines. In some exampleembodiments, the one or more processors or processor-implemented modulesmay be located in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or a combination thereof), registers, or othermachine components that receive, store, transmit, or displayinformation.

As used herein any reference to “one embodiment” or “an embodiment”means that a particular element, feature, structure, or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. The appearances of the phrase “in one embodiment” in variousplaces in the specification are not necessarily all referring to thesame embodiment.

Some embodiments may be described using the expression “coupled” and“connected” along with their derivatives. For example, some embodimentsmay be described using the term “coupled” to indicate that two or moreelements are in direct physical or electrical contact. The term“coupled,” however, may also mean that two or more elements are not indirect contact with each other, but yet still co-operate or interactwith each other. The embodiments are not limited in this context.

As used herein, the terms “comprises,” “comprising,” “includes,”“including,” “has,” “having” or any other variation thereof, areintended to cover a non-exclusive inclusion. For example, a process,method, article, or apparatus that comprises a list of elements is notnecessarily limited to only those elements but may include otherelements not expressly listed or inherent to such process, method,article, or apparatus. Further, unless expressly stated to the contrary,“or” refers to an inclusive or and not to an exclusive or. For example,a condition A or B is satisfied by any one of the following: A is true(or present) and B is false (or not present), A is false (or notpresent) and B is true (or present), and both A and B are true (orpresent).

In addition, use of the “a” or “an” are employed to describe elementsand components of the embodiments herein. This is done merely forconvenience and to give a general sense of the description. Thisdescription, and the claims that follow, should be read to include oneor at least one and the singular also includes the plural unless it isobvious that it is meant otherwise.

This detailed description is to be construed as an example only and doesnot describe every possible embodiment, as describing every possibleembodiment would be impractical, if not impossible. One could implementnumerous alternate embodiments, using either current technology ortechnology developed after the filing date of this application.

What is claimed:
 1. A computer-implemented method comprising: receivinga first hematoxylin and eosin (H&E) slide prepared from a tumor block atan image-based nucleic acid yield prediction system having one or moreprocessors; identifying, using the one or more processors, tumor cellsof the H&E slide from a digital image; generating a tumor area maskbased at least in part on the identified tumor cells, wherein the tumorarea mask defines a dissection boundary of the H&E slide associated withthe digital image; predicting an expected yield of nucleic acid for thetumor cells within the dissection boundary; based on the predictedexpected yield of nucleic acid, identifying a number of H&E slides toprepare from the tumor block to satisfy a total nucleic yield; andpreparing the number of H&E slides from the tumor block when thepredicted expected yield of nucleic acid exceeds a predeterminedthreshold.
 2. The method of claim 1, wherein the predetermined thresholdis based at least in part on one or more specification requirements. 3.The method of claim 2, wherein specification requirements are based on aminimum surface area within the dissection boundary.
 4. The method ofclaim 2, wherein specification requirements are based on a minimum cellcount within the dissection boundary.
 5. The method of claim 2, whereinspecification requirements are based on a minimum cell density withinthe dissection boundary.
 6. The method of claim 2, wherein specificationrequirements are set by an entity.
 7. The method of claim 1, wherein thepredetermined threshold is selected from a range between and including50 ng-2000 ng.
 8. The method of claim 1, wherein predicting the expectedyield of nucleic acid for the tumor cells within the dissection boundarycomprises: providing the digital image of the H&E slide to a modeltrained on a plurality of H&E slides having labeled dissectionboundaries and labeled total nucleic yield.
 9. The method of claim 1,wherein predicting the expected yield of nucleic acid for the tumorcells within the dissection boundary comprises: counting the number oftumor cells identified within the dissection boundary and multiplyingthe count by a tumor cell average nucleic acid yield.
 10. The method ofclaim 1, wherein predicting the expected yield of nucleic acid for thetumor cells within the dissection boundary comprises: calculating thesurface area of the dissection boundary and multiplying the surface areaby a dissection boundary average nucleic acid yield.
 11. The method ofclaim 1, wherein the dissection boundary is a microdissection boundary.12. The method of claim 1, wherein the dissection boundary is amacrodissection boundary.
 13. The method of claim 1, wherein thedissection boundary is a whole slide boundary.
 14. The method of claim1, wherein identifying the tumor cells of the H&E slide from the digitalimage comprises identifying the tumor cells using a trained cellsegmentation model.
 15. The method of claim 14, wherein identifying thetumor using the trained cell segmentation model comprises: applying,using the one or more processors, a plurality of tile images formed fromthe digital image to the trained cell segmentation model and, for eachtile, assigning a cell classification to one or more pixels within thetile image.
 16. The method of claim 15, wherein assigning the cellclassification to one or more pixels within the tile image comprises:identifying, using the one or more processors, the one or more pixels asa cell interior, a cell border, or a cell exterior and classifying theone or more pixels as the cell interior, the cell border, or the cellexterior.
 17. The method of claim 14, wherein the trained cellsegmentation model is a pixel-resolution three-dimensional UNetclassification model trained to classify a cell interior, a cell border,and a cell exterior.
 18. The method of claim 14, wherein identifying thetumor cells using the trained cell segmentation model comprises:applying, using the one or more processors, each of a plurality of tileimages formed from the digital image to the trained cell segmentationmodel; and performing, using the one or more processors, a registrationon segmented cells in each of the tile images by determining a cellborder of each cell, determining a centroid of each cell, and shiftingcoordinates of centroids to a universal coordinate space for the digitalimage.
 19. The method of claim 14, wherein the trained cell segmentationmodel is trained using a set of H&E slide training images annotated withidentified cell borders, identified cell interiors, and identified cellexteriors.
 20. The method of claim 1, further comprising superimposing,using a viewer, the tumor area mask over the digital image to visuallyindicate to a user which tumor cells to scrape.
 21. The method of claim1, wherein generating the tumor area mask further comprises providingthe digital image of the H&E slide to a model trained on a plurality ofH&E slides having dissection labels.
 22. A system comprising a pathologyslide scanner system configured to perform the method of claim
 1. 23.The system of claim 22, further configured to generate a digital imageof each of the one or more H&E slides from the tumor block.
 24. Thesystem of claim 23, wherein the pathology slide scanner system iscommunicatively coupled to an image-based tumor cells prediction systemthrough a communication network for providing the one or more of digitalimages.
 25. The method of claim 1, wherein the one or more processorsare one or more graphics processing units (GPUs), tensor processingunits (TPUs), and/or central processing units (CPUs).
 26. The method ofclaim 1, wherein the expected yield of nucleic acid is confirmed throughsequencing the tumor cells within the dissection boundary.
 27. Themethod of claim 26, where sequencing is next-generation sequencing. 28.The method of claim 26, where sequencing is short-read sequencing.